LustreError: 613544:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 613544:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 11 PID: 613544 Comm: ll_sa_613348 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 11132:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 11132:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11132:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 11132:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11132:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11132:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 5749:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e018a1af700 x1879929585272320/t4294972699(0) o101->9182c8ea-a723-481b-ab4b-3dc982f55ee0@0@lo:259/0 lens 376/864 e 0 to 0 dl 1792840624 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 8095:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8095:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 8095:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8095:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8095:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8095:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8095:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8095:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8095:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8095:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8095:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8095:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 17812:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9e01de1c5000: inode [0x200000401:0x33b:0x0] mdc close failed: rc = -13
Lustre: ll_ost_io00_004: service thread pid 8338 was inactive for 41.502 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_002 state:D
task:ll_ost_io00_000 state:R running task stack:0 pid:6525 ppid:2 flags:0x80004080
Call Trace:
? kfree+0x28a/0x380
? osd_trunc_unlock_all+0x68/0x1e0 [osd_ldiskfs]
Lustre: Skipped 2 previous similar messages
task:ll_ost_io00_004 state:R running task stack:0 pid:8338 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? ___might_sleep+0x69/0xf0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
__ldiskfs_journal_get_write_access+0x48/0xd0 [ldiskfs]
? osd_fallocate_preallocate.isra.38+0x452/0xc30 [osd_ldiskfs]
ldiskfs_reserve_inode_write+0xbc/0x110 [ldiskfs]
? __might_sleep+0x60/0xc0
ldiskfs_mark_inode_dirty+0xa6/0x390 [ldiskfs]
osd_fallocate_preallocate.isra.38+0x452/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
osd_fallocate+0x151/0x500 [osd_ldiskfs]
ofd_object_fallocate+0x83a/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
running task stack:0 pid:6527 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
? preempt_schedule_common+0x2f/0x60
? _cond_resched+0x3a/0x50
? osd_trans_stop+0x529/0xe10 [osd_ldiskfs]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x99a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
2[22541]: segfault at 8 ip 00007f1d1b39a875 sp 00007ffefd5f5cf0 error 4 in ld-2.28.so[7f1d1b379000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[22893]: segfault at 8 ip 00007f5dc7ad0875 sp 00007ffe353e7910 error 4 in ld-2.28.so[7f5dc7aaf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 7 previous similar messages
0[25525]: segfault at 8 ip 00007faf13847875 sp 00007fff4feadb30 error 4 in ld-2.28.so[7faf13826000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 13349:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 13349:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 13349:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 13349:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 13349:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 13349:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 13349:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 13349:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 13349:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 13349:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 13349:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 13349:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: ll_ost_io00_006: service thread pid 10526 was inactive for 74.585 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: 8095:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8095:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 8095:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8095:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8095:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8095:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8095:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8095:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8095:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8095:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8095:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8095:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 11 previous similar messages
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000cf5e34bc/0xf129bb4b302c9515 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x11:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 20480->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b302c950e expref: 3328 pid: 8229 timeout: 178 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff9e019316a800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff9e019316a800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff9e019316a800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 34847:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff9e019316a800: namespace resource [0x280000400:0x11:0x0].0x0 (ffff9e0185dcd500) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_004: service thread pid 8338 completed after 105.295s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
hrtimer: interrupt took 33510854 ns
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8339:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 27 previous similar messages
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8339:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8339:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8339:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8339:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 27 previous similar messages
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8339:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 27 previous similar messages
17[40839]: segfault at 564188b0a000 ip 0000564188b0a000 sp 00007ffc7ae01cd8 error 14 in 17[564188d0a000+1000]
Code: Unable to access opcode bytes at RIP 0x564188b09fd6.
INFO: task fallocate:9667 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:9667 ppid:7835 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f95e909c70b
Code: Unable to access opcode bytes at RIP 0x7f95e909c6e1.
RSP: 002b:00007fff84b3ef78 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f95e909c70b
RDX: 0000000000003f58 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004bcd R11: 0000000000000246 R12: 00007fff84b3f198
R13: 0000000000000001 R14: 00007fff84b3efd8 R15: 0000000000003f58
INFO: task fallocate:10577 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:10577 ppid:7671 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f116e42770b
Code: Unable to access opcode bytes at RIP 0x7f116e4276e1.
RSP: 002b:00007ffef82b0b98 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f116e42770b
RDX: 0000000000005f8c RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005e2d R11: 0000000000000246 R12: 00007ffef82b0db8
R13: 0000000000000001 R14: 00007ffef82b0bf8 R15: 0000000000005f8c
INFO: task fallocate:12421 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:12421 ppid:7759 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f57a3a5b70b
Code: Unable to access opcode bytes at RIP 0x7f57a3a5b6e1.
RSP: 002b:00007ffee4460ca8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f57a3a5b70b
RDX: 0000000000002ea1 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000cdf R11: 0000000000000246 R12: 00007ffee4460ec8
R13: 0000000000000000 R14: 00007ffee4460d08 R15: 0000000000002ea1
Lustre: 13349:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 549, rollback = 7
Lustre: 13349:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 25 previous similar messages
Lustre: 13349:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 13349:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 13349:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 13349:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 13349:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/549/0, punch: 0/0/0, quota 1/3/0
Lustre: 13349:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 13349:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 13349:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 13349:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 13349:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 25 previous similar messages
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000292dadb3/0xf129bb4b304347e0 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xda:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b304347d9 expref: 19865 pid: 30673 timeout: 316 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff9e019316a800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff9e019316a800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff9e019316a800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3045:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000401:0x11b1:0x0]/ may get corrupted (rc -108)
Lustre: 3044:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x10f6:0x0]/ may get corrupted (rc -108)
Lustre: 3047:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000401:0x11da:0x0]/ may get corrupted (rc -108)
Lustre: 3045:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x11c1:0x0]/ may get corrupted (rc -108)
LustreError: 62482:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff9e019316a800: namespace resource [0x280000400:0xda:0x0].0x0 (ffff9e0184282000) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
Lustre: 8157:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 5761:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5761:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 65466:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9e019316a800: inode [0x200000402:0x12b2:0x0] mdc close failed: rc = -13
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 00000000032d1a18/0xf129bb4b30496bc9 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x110:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->20479) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b30496b83 expref: 15 pid: 17645 timeout: 357 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: lustre-OST0000-osc-ffff9e019316a800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff9e019316a800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3057:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x141c:0x0]/ may get corrupted (rc -108)
Lustre: 3044:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x13fa:0x0]/ may get corrupted (rc -108)
Lustre: 3043:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x140e:0x0]/ may get corrupted (rc -108)
Lustre: 3046:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x1433:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_002: service thread pid 6527 completed after 286.994s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
INFO: task fallocate:9667 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:9667 ppid:7835 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f95e909c70b
Code: Unable to access opcode bytes at RIP 0x7f95e909c6e1.
RSP: 002b:00007fff84b3ef78 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f95e909c70b
RDX: 0000000000003f58 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004bcd R11: 0000000000000246 R12: 00007fff84b3f198
R13: 0000000000000001 R14: 00007fff84b3efd8 R15: 0000000000003f58
INFO: task fallocate:12421 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:12421 ppid:7759 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f57a3a5b70b
Code: Unable to access opcode bytes at RIP 0x7f57a3a5b6e1.
RSP: 002b:00007ffee4460ca8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f57a3a5b70b
RDX: 0000000000002ea1 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000cdf R11: 0000000000000246 R12: 00007ffee4460ec8
R13: 0000000000000000 R14: 00007ffee4460d08 R15: 0000000000002ea1
INFO: task fallocate:42143 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:42143 ppid:7742 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f8724b0070b
Code: Unable to access opcode bytes at RIP 0x7f8724b006e1.
RSP: 002b:00007ffeb249b9c8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f8724b0070b
RDX: 0000000000002be5 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006bdd R11: 0000000000000246 R12: 00007ffeb249bbe8
R13: 0000000000000000 R14: 00007ffeb249ba28 R15: 0000000000002be5
INFO: task chmod:42568 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:42568 ppid:7701 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x134f/0x1850 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fba7fbfa258
Code: Unable to access opcode bytes at RIP 0x7fba7fbfa22e.
RSP: 002b:00007ffdf45ad758 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fba7fbfa258
RDX: 00000000000006d7 RSI: 00005597fc13c500 RDI: 00000000ffffff9c
RBP: 00005597fc13c470 R08: 0000000000000000 R09: 00000000000006d7
R10: 0000000000000fff R11: 0000000000000246 R12: 00005597fa9b1bb4
R13: 00005597fc13c500 R14: 00005597fc13d730 R15: 0000000000008000
INFO: task chmod:42572 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:42572 ppid:7796 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x134f/0x1850 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fcc4b315258
Code: Unable to access opcode bytes at RIP 0x7fcc4b31522e.
RSP: 002b:00007fff9b743478 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fcc4b315258
RDX: 0000000000000b82 RSI: 0000561f1971c500 RDI: 00000000ffffff9c
RBP: 0000561f1971c470 R08: 0000000000000000 R09: 0000000000000b82
R10: 0000000000000fff R11: 0000000000000246 R12: 0000561f190adbb4
R13: 0000561f1971c500 R14: 0000561f1971d730 R15: 0000000000008000
Lustre: lustre-OST0000-osc-ffff9e019316a800: disconnect after 20s idle
Lustre: mdt00_012: service thread pid 8514 was inactive for 43.429 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 6 previous similar messages
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000071443a7b/0xf129bb4b305732bf lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x180:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 12288->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b305732b8 expref: 29011 pid: 30673 timeout: 439 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: lustre-OST0001-osc-ffff9e019316a800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff9e019316a800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 69447:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff9e019316a800: namespace resource [0x240000400:0x180:0x0].0x0 (ffff9e018415b700) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
LustreError: 7933:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 0000000015716282 ns: mdt-lustre-MDT0000_UUID lock: 00000000dd724f1d/0xf129bb4b3058e097 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x141f:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xf129bb4b3058e058 expref: 521 pid: 7933 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: mdt00_013: service thread pid 8524 completed after 99.917s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_000: service thread pid 5760 completed after 99.951s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9e01de1c5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-MDT0000-mdc-ffff9e01de1c5000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_012: service thread pid 8514 completed after 99.721s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 7974 completed after 99.802s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 7933 completed after 99.724s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 69279:0:(statahead.c:2479:start_statahead_thread()) lustre: cannot start ll_sa_69279 thread for 'racer': rc = -4
Lustre: mdt_out00_000: service thread pid 5752 completed after 99.971s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 69388:0:(file.c:6392:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000402:0x141f:0x0] error: rc = -5
Lustre: mdt00_003: service thread pid 6967 completed after 99.741s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 8533 completed after 99.853s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 69450:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9e01de1c5000: inode [0x200000401:0x13bf:0x0] mdc close failed: rc = -108
Lustre: 28616:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 28616:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 28616:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 28616:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 42 previous similar messages
Lustre: 28616:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 28616:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 42 previous similar messages
Lustre: 28616:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 28616:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 42 previous similar messages
Lustre: 28616:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 28616:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 42 previous similar messages
Lustre: 28616:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 28616:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 42 previous similar messages
Lustre: 5762:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
ptlrpc_watchdog_fire: 9 callbacks suppressed
Lustre: ll_ost00_010: service thread pid 17654 was inactive for 40.738 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost00_010 state:I stack:0 pid:17654 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
task:ll_ost00_005 state:I
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc27/0x1100 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? lu_object_find_at+0x160/0xb80 [obdclass]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? ldlm_blocking_ast_nocheck+0x3f0/0x3f0 [ptlrpc]
ofd_destroy_by_fid+0x391/0x7c0 [ofd]
? ldlm_blocking_ast_nocheck+0x3f0/0x3f0 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ofd_destroy_hdl+0x34a/0xfd0 [ofd]
? ofd_destroy_hdl+0x34a/0xfd0 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
task:ll_ost00_006 state:I stack:0 pid:8229 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc27/0x1100 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? lu_object_find_at+0x160/0xb80 [obdclass]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
stack:0 pid:8224 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc27/0x1100 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
? ldlm_blocking_ast_nocheck+0x3f0/0x3f0 [ptlrpc]
ofd_destroy_by_fid+0x391/0x7c0 [ofd]
? ldlm_blocking_ast_nocheck+0x3f0/0x3f0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? lu_object_find_at+0x160/0xb80 [obdclass]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? ldlm_blocking_ast_nocheck+0x3f0/0x3f0 [ptlrpc]
ofd_destroy_by_fid+0x391/0x7c0 [ofd]
? ldlm_blocking_ast_nocheck+0x3f0/0x3f0 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ofd_destroy_hdl+0x34a/0xfd0 [ofd]
? ofd_destroy_hdl+0x34a/0xfd0 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ofd_destroy_hdl+0x34a/0xfd0 [ofd]
? ofd_destroy_hdl+0x34a/0xfd0 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
INFO: task fallocate:9667 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:9667 ppid:7835 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f95e909c70b
Code: Unable to access opcode bytes at RIP 0x7f95e909c6e1.
RSP: 002b:00007fff84b3ef78 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f95e909c70b
RDX: 0000000000003f58 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004bcd R11: 0000000000000246 R12: 00007fff84b3f198
R13: 0000000000000001 R14: 00007fff84b3efd8 R15: 0000000000003f58
INFO: task fallocate:12421 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:12421 ppid:7759 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f57a3a5b70b
Code: Unable to access opcode bytes at RIP 0x7f57a3a5b6e1.
RSP: 002b:00007ffee4460ca8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f57a3a5b70b
RDX: 0000000000002ea1 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000cdf R11: 0000000000000246 R12: 00007ffee4460ec8
R13: 0000000000000000 R14: 00007ffee4460d08 R15: 0000000000002ea1
Lustre: 8157:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 8157:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
19[93738]: segfault at 0 ip 0000563a6662cb47 sp 00007ffcec5a5c60 error 6 in 19[563a66628000+7000]
Code: Unable to access opcode bytes at RIP 0x563a6662cb1d.
Lustre: 8157:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 8157:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000001c98ab3d/0xf129bb4b30439144 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0xdf:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8192->40959) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0xf129bb4b30439113 expref: 1216 pid: 6518 timeout: 547 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff9e01de1c5000: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-OST0000-osc-ffff9e01de1c5000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: ll_ost00_010: service thread pid 17654 completed after 102.213s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_006: service thread pid 8229 completed after 103.166s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_006: service thread pid 10526 completed after 463.785s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-OST0000-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3051:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x9a6:0x0]/ may get corrupted (rc -108)
Lustre: ll_ost00_005: service thread pid 8224 completed after 103.152s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 3051:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x1cfa:0x0]/ may get corrupted (rc -108)
Lustre: 3058:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0xa07:0x0]// may get corrupted (rc -108)
Lustre: 3043:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x946:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff9e01de1c5000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: ll_ost_io00_000: service thread pid 6525 completed after 473.866s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 8179:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792841074 with bad export cookie 17377626568790844387
LustreError: lustre-OST0002-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3047:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x1e8e:0x0]/ may get corrupted (rc -108)
Lustre: 3048:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x9f3:0x0]/ may get corrupted (rc -108)
LustreError: 99008:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff9e01de1c5000: namespace resource [0x300000400:0x1e9:0x0].0x0 (ffff9e01f18d7f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 99008:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 8157:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000005bb7d3d7/0xf129bb4b305dc202 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x1bb:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8192->20479) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xf129bb4b305dc0d5 expref: 6051 pid: 6973 timeout: 575 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffff9e019316a800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff9e019316a800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffff9e019316a800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3046:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x20c8:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
5[107795]: segfault at 0 ip 00005571c4509b47 sp 00007fff729f9af0 error 6 in 12[5571c4505000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000f66fe909/0xf129bb4b3062e8be lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x1e9:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->12287) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0xf129bb4b3062e840 expref: 732 pid: 55525 timeout: 595 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6517:0:(client.c:1411:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e0222208700 x1879929655642240/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff9e019316a800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff9e019316a800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff9e019316a800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3048:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0xcc8:0x0]/ may get corrupted (rc -108)
Lustre: 3045:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x228a:0x0]// may get corrupted (rc -108)
Lustre: 3046:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0xd95:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
Lustre: 5761:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5761:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
13[120919]: segfault at 0 ip 000055f552d20b47 sp 00007fff2e6dbe50 error 6 in 2[55f552d1c000+7000]
Code: Unable to access opcode bytes at RIP 0x55f552d20b1d.
LustreError: 132615:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9e019316a800: inode [0x200000402:0x2a11:0x0] mdc close failed: rc = -13
LustreError: 132615:0:(file.c:252:ll_close_inode_openhandle()) Skipped 5 previous similar messages
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000005c1dee7/0xf129bb4b30770cae lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x286:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b30770ca7 expref: 43325 pid: 77761 timeout: 683 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff9e019316a800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff9e019316a800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff9e019316a800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 6516:0:(client.c:1411:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9e01b3135780 x1879929674103552/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: 3054:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x2ad0:0x0]// may get corrupted (rc -108)
Lustre: 3056:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x2bca:0x0]/ may get corrupted (rc -108)
Lustre: 3055:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x2bb2:0x0]// may get corrupted (rc -108)
Lustre: 3057:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x2bd4:0x0]/ may get corrupted (rc -108)
Lustre: 3057:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x165c:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff9e019316a800: Connection restored to 0@lo (at 0@lo)
18[141214]: segfault at 8 ip 00007f35c2ecd875 sp 00007ffdb2cc3e70 error 4 in ld-2.28.so[7f35c2eac000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 12648:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12648:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 205 previous similar messages
Lustre: 12648:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12648:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 216 previous similar messages
Lustre: 12648:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12648:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 216 previous similar messages
Lustre: 12648:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 12648:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 216 previous similar messages
Lustre: 12648:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12648:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 216 previous similar messages
Lustre: 12648:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12648:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 216 previous similar messages
Lustre: 5762:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5762:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000207d51f6/0xf129bb4b309695ec lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x37a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8192->16383) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xf129bb4b309695e5 expref: 499 pid: 6973 timeout: 831 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff9e01de1c5000: operation ost_read to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff9e01de1c5000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
LustreError: lustre-OST0003-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3044:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x26b9:0x0]/ may get corrupted (rc -108)
Lustre: 3046:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x26d1:0x0]/ may get corrupted (rc -108)
Lustre: 3057:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x2686:0x0]/ may get corrupted (rc -108)
Lustre: 3056:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x26e3:0x0]/ may get corrupted (rc -108)
Lustre: 3056:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x39e4:0x0]/ may get corrupted (rc -108)
Lustre: 3057:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x26b4:0x0]/ may get corrupted (rc -108)
Lustre: 3053:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x26b5:0x0]// may get corrupted (rc -108)
LustreError: 179541:0:(lcommon_cl.c:187:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000403:0x26b5:0x0]: rc = -108
LustreError: 179541:0:(llite_lib.c:3884:ll_prep_inode()) lustre: new_inode - fatal error: rc = -108
Lustre: 3057:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x25f4:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff9e01de1c5000: Connection restored to 0@lo (at 0@lo)
11[182382]: segfault at 0 ip 000055fd51a11d50 sp 00007ffc7617fd68 error 6 in 19[55fd51a0d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[186364]: segfault at 0 ip 000055a2b3c60d50 sp 00007ffe1fd0e9a8 error 6 in 13[55a2b3c5c000+7000]
13[186385]: segfault at 0 ip 000055a9e229ad50 sp 00007ffcd256e3f8 error 6
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
in 13[55a9e2296000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 5761:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5761:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 2 previous similar messages
LustreError: 201435:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
11[211614]: segfault at 8 ip 00007f1616515875 sp 00007fff3c6098f0 error 4 in ld-2.28.so[7f16164f4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000005361357a/0xf129bb4b30b4d2aa lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x480:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 24576->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b30b4d29c expref: 4710 pid: 77761 timeout: 963 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffff9e01de1c5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-ffff9e01de1c5000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3046:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x44e0:0x0]/ may get corrupted (rc -108)
Lustre: 3054:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x33f4:0x0]// may get corrupted (rc -108)
Lustre: 3049:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x3351:0x0]/ may get corrupted (rc -108)
Lustre: 3050:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x3421:0x0]/ may get corrupted (rc -108)
Lustre: 3052:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x45b7:0x0]/ may get corrupted (rc -108)
Lustre: 3052:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x33ab:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff9e01de1c5000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 5762:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 5762:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 'ost_io' is processing requests too slowly, client may timeout. Late by 25s, missed 0 early replies (reqs waiting=0 active=5, at_estimate=491, delay=7ms)
LustreError: lustre-OST0001-osc-ffff9e01de1c5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-OST0001-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3047:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x5153:0x0]/ may get corrupted (rc -108)
Lustre: 3051:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x405f:0x0]/ may get corrupted (rc -108)
Lustre: 3050:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x3f7e:0x0]/ may get corrupted (rc -108)
Lustre: 3054:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x4056:0x0]// may get corrupted (rc -108)
LustreError: lustre-OST0003-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3055:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x405c:0x0]/ may get corrupted (rc -108)
Lustre: 3058:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x3fc7:0x0]/ may get corrupted (rc -108)
14[261580]: segfault at 8 ip 00007f0b607cc875 sp 00007ffcd6108260 error 4 in ld-2.28.so[7f0b607ab000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6527:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6527:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 265 previous similar messages
Lustre: 6527:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6527:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 275 previous similar messages
Lustre: 6527:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6527:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 275 previous similar messages
Lustre: 6527:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6527:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 275 previous similar messages
Lustre: 6527:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6527:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 275 previous similar messages
Lustre: 6527:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6527:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 275 previous similar messages
Lustre: 67233:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/-14), not sending early reply req@ffff9e01c9fe7000 x1879929658286848/t0(0) o22->9182c8ea-a723-481b-ab4b-3dc982f55ee0@0@lo:634/0 lens 440/432 e 0 to 0 dl 1792841754 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000019633025/0xf129bb4b30e6e069 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x612:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 16384->49151) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xf129bb4b30e6e05b expref: 14859 pid: 6516 timeout: 1228 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5738:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
Lustre: lustre-OST0001-osc-ffff9e01de1c5000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: lustre-OST0001-osc-ffff9e01de1c5000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3055:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000403:0x4882:0x0]/ may get corrupted (rc -108)
Lustre: 3055:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.17@tcp:/lustre/fid: [0x200000402:0x595f:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff9e01de1c5000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 2 previous similar messages
17[278269]: segfault at 8 ip 00007fe80334c875 sp 00007ffc3a9dd3c0 error 4 in ld-2.28.so[7fe80332b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[280912]: segfault at 8 ip 00007f96b3353875 sp 00007ffc02e68f60 error 4 in ld-2.28.so[7f96b3332000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 10526:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-13), not sending early reply req@ffff9e01deaa0000 x1879929674282752/t0(0) o22->9182c8ea-a723-481b-ab4b-3dc982f55ee0@0@lo:708/0 lens 440/432 e 0 to 0 dl 1792841828 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
8[292565]: segfault at 0 ip 0000557f798aeb47 sp 00007ffefd4a1830 error 6 in 8[557f798aa000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14[294101]: segfault at 8 ip 00007f3875b80875 sp 00007fff4205af60 error 4 in ld-2.28.so[7f3875b5f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 94001:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 94001:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 4 previous similar messages
7[305154]: segfault at 8 ip 00007fe57c7aa875 sp 00007fffeae98280 error 4 in ld-2.28.so[7fe57c789000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[308848]: segfault at 0 ip 000056150af70b47 sp 00007ffc13099900 error 6 in 9[56150af6c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 13349:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (3/-15), not sending early reply req@ffff9e01d546c280 x1879929708056704/t0(0) o22->26e31d16-2df6-4feb-9415-91e033f13394@0@lo:107/0 lens 440/432 e 0 to 0 dl 1792841982 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
16[325837]: segfault at 8 ip 00007f46375e3875 sp 00007ffe316f21f0 error 4 in ld-2.28.so[7f46375c2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3048:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792841136/real 1792841136] req@ffff9e01c9fe4d00 x1879929658286848/t0(0) o22->lustre-OST0002-osc-ffff9e019316a800@0@lo:6/4 lens 440/432 e 0 to 1 dl 1792841759 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0002: Client 9182c8ea-a723-481b-ab4b-3dc982f55ee0 (at 0@lo) reconnecting
8[358410]: segfault at 8 ip 00007fd7ee605875 sp 00007ffc10258640 error 4 in ld-2.28.so[7fd7ee5e4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: ll_ost_io00_005: service thread pid 8339 was inactive for 998.279 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 2 previous similar messages
task:ll_ost_io00_005 state:R running task stack:0 pid:8339 ppid:2 flags:0x80004080
Call Trace:
? do_raw_spin_unlock+0x79/0x190
? slab_post_alloc_hook+0x6d/0x390
? debug_check_no_obj_freed+0x16f/0x2e5
? osd_write+0x34d/0x650 [osd_ldiskfs]
? schedule+0xc4/0x180
? wait_transaction_locked+0xab/0x110
? ldiskfs_ext_map_blocks+0x1a7/0x1770 [ldiskfs]
? __mod_timer+0x395/0x690
? __might_sleep+0x60/0xc0
? ldiskfs_map_blocks+0x2ac/0x920 [ldiskfs]
? osd_fallocate_preallocate.isra.38+0x396/0xc30 [osd_ldiskfs]
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? ofd_trans_create+0x7f/0x2b0 [ofd]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
9[370689]: segfault at 8 ip 00007f5a232d8875 sp 00007fff34795ab0 error 4 in ld-2.28.so[7f5a232b7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: ll_ost_io00_003: service thread pid 8095 was inactive for 990.298 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_003 state:R running task stack:0 pid:8095 ppid:2 flags:0x80004080
Call Trace:
? ldiskfs_map_blocks+0x2ac/0x920 [ldiskfs]
? osd_fallocate_preallocate.isra.38+0x396/0xc30 [osd_ldiskfs]
? dt_txn_hook_start+0xc0/0xf0 [obdclass]
? osd_trans_start+0x175/0x9e0 [osd_ldiskfs]
? dt_trans_start+0x20/0x70 [obdclass]
? ofd_trans_start+0x7d/0xf0 [ofd]
? ofd_object_fallocate+0x6cf/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
7[388386]: segfault at 55e8aab59f24 ip 000055e8aab59f24 sp 00007ffd6cb0f6e8 error 7 in 7[55e8aab55000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: ll_ost_io00_013: service thread pid 214588 was inactive for 548.775 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_013 state:D stack:0 pid:214588 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
19[396971]: segfault at 1b30 ip 0000000000001b30 sp 00007ffdfedd2ae0 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
Lustre: 'ost_io' is processing requests too slowly, client may timeout. Late by 14s, missed 1 early replies (reqs waiting=0 active=6, at_estimate=600, delay=1ms)
Lustre: 45749:0:(service.c:1570:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-14s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff9e01af001880 x1879929769317888/t0(0) o22->26e31d16-2df6-4feb-9415-91e033f13394@0@lo:442/0 lens 440/432 e 1 to 0 dl 1792842317 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 5761:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 5761:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 383 previous similar messages
Lustre: 5761:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0
Lustre: 5761:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 383 previous similar messages
Lustre: 5761:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 7/249/0
Lustre: 5761:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 383 previous similar messages
Lustre: 5761:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 7/117/1, delete: 3/6/1
Lustre: 5761:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 383 previous similar messages
Lustre: 5761:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 5761:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 383 previous similar messages
Lustre: 6526:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6526:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 367 previous similar messages
Lustre: ll_ost_io00_007: service thread pid 11132 was inactive for 999.727 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 416057:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9e01de1c5000: inode [0x200000403:0x790c:0x0] mdc close failed: rc = -13
LustreError: 416057:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
8[425674]: segfault at 0 ip 00005645dc2b2b47 sp 00007fff5a24ca50 error 6 in 8[5645dc2ae000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5[434591]: segfault at 8 ip 00007f3d21c3e875 sp 00007ffc87c25b20 error 4 in ld-2.28.so[7f3d21c1d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3048:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792841717/real 1792841717] req@ffff9e01d81a7700 x1879929769317888/t0(0) o22->lustre-OST0000-osc-ffff9e01de1c5000@0@lo:6/4 lens 440/432 e 1 to 1 dl 1792842322 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0000-osc-ffff9e01de1c5000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0000: Client 26e31d16-2df6-4feb-9415-91e033f13394 (at 0@lo) reconnecting
Lustre: lustre-OST0000-osc-ffff9e01de1c5000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: 6526:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (2/-153), not sending early reply req@ffff9e0210968a80 x1879929783596544/t0(0) o22->26e31d16-2df6-4feb-9415-91e033f13394@0@lo:675/0 lens 440/432 e 0 to 0 dl 1792842550 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
11[486062]: segfault at 0 ip 0000556fe5cedb47 sp 00007ffe7288fe70 error 6 in 11[556fe5ce9000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[487637]: segfault at 0 ip 00005571487efb47 sp 00007ffcc563c4b0 error 6 in 10[5571487eb000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 3047:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792841210/real 1792841210] req@ffff9e01d5bae580 x1879929674282752/t0(0) o22->lustre-OST0000-osc-ffff9e019316a800@0@lo:6/4 lens 440/432 e 0 to 1 dl 1792841833 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0000: Client 9182c8ea-a723-481b-ab4b-3dc982f55ee0 (at 0@lo) reconnecting
Lustre: 5760:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 5760:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 16 previous similar messages
10[506560]: segfault at 0 ip 00005635eed4bb47 sp 00007ffe10079cb0 error 6 in 10[5635eed47000+7000]
Code: Unable to access opcode bytes at RIP 0x5635eed4bb1d.
8[517176]: segfault at 8 ip 00007f98cd806875 sp 00007ffeb237ddb0 error 4 in ld-2.28.so[7f98cd7e5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[549008]: segfault at 8 ip 00007f7f75b2b875 sp 00007fffc03ff7c0 error 4 in ld-2.28.so[7f7f75b0a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6527:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6527:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 314 previous similar messages
Lustre: 6527:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6527:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 314 previous similar messages
Lustre: 6527:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/525/0, punch: 0/0/0, quota 1/3/0
Lustre: 6527:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 314 previous similar messages
Lustre: 6527:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6527:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 314 previous similar messages
Lustre: 6527:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6527:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 314 previous similar messages
Lustre: 12648:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 537, rollback = 7
Lustre: 12648:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 307 previous similar messages
2[553234]: segfault at 8 ip 00007fde298f4875 sp 00007ffec712bf80 error 4 in ld-2.28.so[7fde298d3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ptlrpc_watchdog_fire: 1 callbacks suppressed
Lustre: ll_ost_io00_000: service thread pid 6525 was inactive for 1224.106 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_000 state:R running task stack:0 pid:6525 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
5[569792]: segfault at 8 ip 00007f087757f875 sp 00007ffe63e13d80 error 4 in ld-2.28.so[7f087755e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[570042]: segfault at 8 ip 00007f0a13816875 sp 00007ffcfd718a60 error 4 in ld-2.28.so[7f0a137f5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[571282]: segfault at 8 ip 00007fb705da5875 sp 00007fff70f74ba0 error 4 in ld-2.28.so[7fb705d84000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[572343]: segfault at 0 ip 00005639c7f8bde0 sp 00007ffd364c0008 error 6 in 7[5639c7f87000+7000]
Code: 48 83 f8 ff 75 b4 48 89 df 5b 5d 41 5c e9 58 c8 ff ff 0f 1f 84 00 00 00 00 00 44 89 65 00 b8 ff ff ff ff eb b9 0f 1f 44 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5[592660]: segfault at 0 ip 000055cea8ddbb47 sp 00007fff05b17d10 error 6 in 14 (deleted)[55cea8dd7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

LustreError: 299018:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 299018:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 7 PID: 299018 Comm: ll_sa_298860 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 7867:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa0a26a421f80 x1879812599851136/t4294967387(0) o101->6687a0fe-0119-46ad-8fd8-b43e8afcd5ec@0@lo:418/0 lens 376/840 e 0 to 0 dl 1792729043 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 6574:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6574:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6574:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6574:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6574:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6574:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6574:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6574:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6574:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6574:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6574:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6574:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6574:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6574:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6574:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6574:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6574:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6574:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6572:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6572:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 6572:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6572:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6572:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6572:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6572:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6572:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6572:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6572:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6572:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6572:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12518:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12518:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12518:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12518:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 12518:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12518:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12518:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12518:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12518:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12518:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 12518:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12518:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12518:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12518:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: ll_ost_io00_001: service thread pid 6573 was inactive for 43.748 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_001 state:R running task stack:0 pid:6573 ppid:2 flags:0x80004080
Call Trace:
? 0xffffffffb4c00000
? __change_page_attr_set_clr+0x75/0x3b0
? __alloc_pages_nodemask+0x22a/0x550
task:ll_ost_io00_002 state:R running task stack:0 pid:6574 ppid:2 flags:0x80004080
? pick_next_task_fair+0x53/0x6b0
Call Trace:
? get_partial_node+0x1bb/0x780
__schedule+0x351/0xcb0
? __find_get_block+0x118/0x540
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? __might_sleep+0x60/0xc0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
slab_pre_alloc_hook.constprop.64+0x138/0x1f0
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
kmem_cache_alloc_trace+0x5b/0x440
? osd_declare_inode_qid+0x48f/0x6c0 [osd_ldiskfs]
osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
osd_declare_fallocate+0x4ab/0x740 [osd_ldiskfs]
dt_declare_fallocate+0x30/0xa0 [obdclass]
ofd_object_fallocate+0x485/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ldiskfs_do_update_inode+0x79a/0xbb0 [ldiskfs]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? jbd2_journal_get_write_access+0x49/0xf0
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? __find_get_block+0x118/0x540
? __might_sleep+0x60/0xc0
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? apic_timer_interrupt+0xa/0x20
? apic_timer_interrupt+0xa/0x20
? __pv_queued_spin_lock_slowpath+0x182/0x6f0
? do_raw_spin_lock+0x17c/0x190
? _raw_spin_lock+0x25/0x30
? osd_trans_stop+0x383/0xe10 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 6572:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6572:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 6572:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6572:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6572:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6572:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6572:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6572:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6572:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6572:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6572:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6572:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8678:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8678:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 8678:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8678:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8678:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8678:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8678:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8678:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8678:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8678:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8678:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8678:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: ll_ost_io00_005: service thread pid 11979 was inactive for 74.671 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
task:ll_ost_io00_005 state:R running task stack:0 pid:11979 ppid:2 flags:0x80004080
Call Trace:
? slab_free_freelist_hook.isra.49+0x111/0x230
? __might_sleep+0x60/0xc0
? ldiskfs_map_blocks+0x2ac/0x920 [ldiskfs]
? osd_fallocate_preallocate.isra.38+0x396/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000daca1bac/0xcf90fd3c8b6b2789 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x19:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 8192->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xcf90fd3c8b6b2774 expref: 18 pid: 10174 timeout: 184 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffffa0a2141f2800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffffa0a2141f2800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffffa0a2141f2800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3053:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x6ff:0x0]// may get corrupted (rc -108)
Lustre: 3054:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x6d5:0x0]/ may get corrupted (rc -108)
Lustre: 3055:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x673:0x0]/ may get corrupted (rc -108)
Lustre: 3058:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x726:0x0]// may get corrupted (rc -108)
Lustre: 3058:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x733:0x0]// may get corrupted (rc -108)
Lustre: 3059:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x750:0x0]/ may get corrupted (rc -108)
LustreError: 31090:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffffa0a2141f2800: namespace resource [0x280000400:0x9c:0x0].0x0 (ffffa0a205336f00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0002-osc-ffffa0a2141f2800: Connection restored to 0@lo (at 0@lo)
Lustre: 8093:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8093:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 37 previous similar messages
Lustre: 8093:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8093:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 8093:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8093:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 8093:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8093:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 8093:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8093:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 8093:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8093:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: ll_ost_io00_001: service thread pid 6573 completed after 102.618s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000344825ad/0xcf90fd3c8b6da489 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x2d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->12287) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xcf90fd3c8b6da43c expref: 1059 pid: 8022 timeout: 206 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffffa0a255463800: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffffa0a255463800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffffa0a255463800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3048:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x76c:0x0]/ may get corrupted (rc -108)
Lustre: 3045:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x6ba:0x0]/ may get corrupted (rc -108)
Lustre: 3047:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x888:0x0]/ may get corrupted (rc -108)
Lustre: 3050:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x91e:0x0]/ may get corrupted (rc -108)
Lustre: 3050:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x8b6:0x0]/ may get corrupted (rc -108)
Lustre: 3052:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x8bc:0x0]/ may get corrupted (rc -108)
Lustre: 3052:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x94a:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffffa0a255463800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_005: service thread pid 11979 completed after 107.051s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
18[37894]: segfault at 8 ip 00007f3a8fc8c875 sp 00007fff839fb940 error 4 in ld-2.28.so[7f3a8fc6b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 37894:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a255463800: inode [0x200000401:0x964:0x0] mdc close failed: rc = -13
15[38258]: segfault at 8 ip 00007f9fbc5d7875 sp 00007ffef91d56b0 error 4 in ld-2.28.so[7f9fbc5b6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[38684]: segfault at 8 ip 00007f635599d875 sp 00007ffef4b0c040 error 4 in ld-2.28.so[7f635597c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 38684:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a255463800: inode [0x200000401:0x964:0x0] mdc close failed: rc = -13
2[39142]: segfault at 8 ip 00007fdfd810b875 sp 00007fff781d9b40 error 4 in ld-2.28.so[7fdfd80ea000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:11899 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:11899 ppid:7755 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f03c328f70b
Code: Unable to access opcode bytes at RIP 0x7f03c328f6e1.
RSP: 002b:00007fff304f9e18 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03c328f70b
RDX: 0000000000007512 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001317 R11: 0000000000000246 R12: 00007fff304fa038
R13: 0000000000000001 R14: 00007fff304f9e78 R15: 0000000000007512
Lustre: 6572:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 525, rollback = 7
Lustre: 6572:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 29 previous similar messages
Lustre: 6572:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6572:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 6572:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6572:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 6572:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/525/0, punch: 0/0/0, quota 1/3/0
Lustre: 6572:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 6572:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6572:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 6572:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6572:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 29 previous similar messages
LustreError: 52475:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a255463800: inode [0x200000401:0xe41:0x0] mdc close failed: rc = -13
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000008cda7168/0xcf90fd3c8b7a875e lrc: 3/0,0 mode: PW/PW res: [0x300000400:0xa5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->16383) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xcf90fd3c8b7a8749 expref: 1283 pid: 6902 timeout: 290 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffffa0a255463800: operation ost_punch to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffffa0a255463800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffffa0a255463800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3050:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0xe4f:0x0]/ may get corrupted (rc -108)
Lustre: 3047:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0xf01:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffffa0a255463800: Connection restored to 0@lo (at 0@lo)
8[55666]: segfault at 8 ip 00007f373f234875 sp 00007ffc3f134d30 error 4 in ld-2.28.so[7f373f213000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[56578]: segfault at 0 ip 000055f4a4aedb47 sp 00007ffc02b0e9c0 error 6 in 11[55f4a4ae9000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 5784:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000097268c3c/0xcf90fd3c8b7fb42c lrc: 3/0,0 mode: PW/PW res: [0x240000400:0xd3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->20479) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xcf90fd3c8b7fb281 expref: 2812 pid: 6903 timeout: 321 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffffa0a2141f2800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffffa0a2141f2800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffffa0a2141f2800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3046:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x105c:0x0]/ may get corrupted (rc -108)
Lustre: 3060:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000401:0x1042:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffffa0a2141f2800: Connection restored to 0@lo (at 0@lo)
10[68009]: segfault at 8 ip 00007fdd68fd6875 sp 00007ffed37935f0 error 4 in ld-2.28.so[7fdd68fb5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[68408]: segfault at 0 ip 000055c0e4a0cb47 sp 00007ffd9c523b20 error 6 in 13[55c0e4a08000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
INFO: task fallocate:11899 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:11899 ppid:7755 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f03c328f70b
Code: Unable to access opcode bytes at RIP 0x7f03c328f6e1.
RSP: 002b:00007fff304f9e18 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03c328f70b
RDX: 0000000000007512 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001317 R11: 0000000000000246 R12: 00007fff304fa038
R13: 0000000000000001 R14: 00007fff304f9e78 R15: 0000000000007512
INFO: task chown:35856 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:35856 ppid:7789 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x134f/0x1850 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f88985cafbe
Code: Unable to access opcode bytes at RIP 0x7f88985caf94.
RSP: 002b:00007ffe8cb83698 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffe8cb838d0 RCX: 00007f88985cafbe
RDX: 000000000000232d RSI: 000055f38852bd20 RDI: 00000000ffffff9c
RBP: 000055f38852cf50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000003277 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000055f38852cfc8 R14: 000000000000232d R15: 0000000000003277
INFO: task fallocate:35897 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35897 ppid:7776 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f57f629470b
Code: Unable to access opcode bytes at RIP 0x7f57f62946e1.
RSP: 002b:00007ffdd220c1d8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f57f629470b
RDX: 0000000000006512 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000002a47 R11: 0000000000000246 R12: 00007ffdd220c3f8
R13: 0000000000000001 R14: 00007ffdd220c238 R15: 0000000000006512
INFO: task chmod:36103 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:36103 ppid:7749 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x134f/0x1850 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f43618d9258
Code: Unable to access opcode bytes at RIP 0x7f43618d922e.
RSP: 002b:00007ffcdff8a268 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f43618d9258
RDX: 000000000000052c RSI: 0000562056cc0500 RDI: 00000000ffffff9c
RBP: 0000562056cc0470 R08: 0000000000000000 R09: 000000000000052c
R10: 0000000000000fff R11: 0000000000000246 R12: 00005620560c6bb4
R13: 0000562056cc0500 R14: 0000562056cc1730 R15: 0000000000008000
Lustre: 8093:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8093:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 75 previous similar messages
Lustre: 8093:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8093:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 8093:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8093:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 8093:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8093:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 8093:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8093:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 79 previous similar messages
Lustre: 8093:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8093:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 79 previous similar messages
4[76707]: segfault at 8 ip 00007fb60f400875 sp 00007ffd090c40b0 error 4 in ld-2.28.so[7fb60f3df000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[77437]: segfault at 8 ip 00007f92dd632875 sp 00007ffc7beacbb0 error 4 in ld-2.28.so[7f92dd611000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 87403:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a255463800: inode [0x200000402:0x19a9:0x0] mdc close failed: rc = -13
Lustre: 5784:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5784:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
3[95492]: segfault at 8 ip 00007fcf44a07875 sp 00007fff2e786900 error 4 in ld-2.28.so[7fcf449e6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:11899 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:11899 ppid:7755 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f03c328f70b
Code: Unable to access opcode bytes at RIP 0x7f03c328f6e1.
RSP: 002b:00007fff304f9e18 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f03c328f70b
RDX: 0000000000007512 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001317 R11: 0000000000000246 R12: 00007fff304fa038
R13: 0000000000000001 R14: 00007fff304f9e78 R15: 0000000000007512
INFO: task chown:35856 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:35856 ppid:7789 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x134f/0x1850 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f88985cafbe
Code: Unable to access opcode bytes at RIP 0x7f88985caf94.
RSP: 002b:00007ffe8cb83698 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffe8cb838d0 RCX: 00007f88985cafbe
RDX: 000000000000232d RSI: 000055f38852bd20 RDI: 00000000ffffff9c
RBP: 000055f38852cf50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000003277 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000055f38852cfc8 R14: 000000000000232d R15: 0000000000003277
INFO: task fallocate:35897 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35897 ppid:7776 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f57f629470b
Code: Unable to access opcode bytes at RIP 0x7f57f62946e1.
RSP: 002b:00007ffdd220c1d8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f57f629470b
RDX: 0000000000006512 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000002a47 R11: 0000000000000246 R12: 00007ffdd220c3f8
R13: 0000000000000001 R14: 00007ffdd220c238 R15: 0000000000006512
INFO: task chmod:36103 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:36103 ppid:7749 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x134f/0x1850 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f43618d9258
Code: Unable to access opcode bytes at RIP 0x7f43618d922e.
RSP: 002b:00007ffcdff8a268 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f43618d9258
RDX: 000000000000052c RSI: 0000562056cc0500 RDI: 00000000ffffff9c
RBP: 0000562056cc0470 R08: 0000000000000000 R09: 000000000000052c
R10: 0000000000000fff R11: 0000000000000246 R12: 00005620560c6bb4
R13: 0000562056cc0500 R14: 0000562056cc1730 R15: 0000000000008000
INFO: task fallocate:56047 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:56047 ppid:7707 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xb47/0xd70 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff794e1970b
Code: Unable to access opcode bytes at RIP 0x7ff794e196e1.
RSP: 002b:00007ffdb2efb428 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff794e1970b
RDX: 0000000000004923 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006f01 R11: 0000000000000246 R12: 00007ffdb2efb648
R13: 0000000000000000 R14: 00007ffdb2efb488 R15: 0000000000004923
Lustre: ll_ost_io00_009: service thread pid 19825 was inactive for 290.303 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_009 state:D stack:0 pid:19825 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
17[109096]: segfault at 8 ip 00007f42797e2875 sp 00007ffdb86bbb30 error 4 in ld-2.28.so[7f42797c1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[113017]: segfault at 8 ip 00007f17114fb875 sp 00007ffe3291cad0 error 4 in ld-2.28.so[7f17114da000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 108661:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 108661:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 183 previous similar messages
Lustre: 108661:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 108661:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 185 previous similar messages
Lustre: 108661:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 108661:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 185 previous similar messages
Lustre: 108661:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 108661:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 185 previous similar messages
Lustre: 108661:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 108661:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 185 previous similar messages
Lustre: 108661:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 108661:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 185 previous similar messages
Lustre: 59516:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffffa0a26b4ad780 x1879812602171648/t0(0) o22->5d5ee6e0-2594-47d4-82cd-570e094068c8@0@lo:263/0 lens 440/432 e 24 to 0 dl 1792729643 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 5784:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5784:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
18[170829]: segfault at 0 ip 0000559779046b47 sp 00007ffef54b2b30 error 6 in 6[559779042000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: ll_ost_io00_004: service thread pid 8678 was inactive for 489.072 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_004 state:R stack:0 pid:8678 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
? pick_next_task_fair+0x30c/0x6b0
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 15972:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffffa0a278852d80 x1879812619649024/t0(0) o22->5d5ee6e0-2594-47d4-82cd-570e094068c8@0@lo:385/0 lens 440/432 e 4 to 0 dl 1792729765 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
10[186877]: segfault at 8 ip 00007f5985789875 sp 00007ffe1cd32900 error 4 in ld-2.28.so[7f5985768000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 194314:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a255463800: inode [0x200000402:0x3b4c:0x0] mdc close failed: rc = -13
2[203064]: segfault at 0 ip 000056278c665b47 sp 00007ffecb397750 error 6 in 2[56278c661000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: ll_ost_io00_007: service thread pid 12522 was inactive for 529.582 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_007 state:D stack:0 pid:12522 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
16[206154]: segfault at 8 ip 00007f5b9f07d875 sp 00007fffd29d48b0 error 4 in ld-2.28.so[7f5b9f05c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3060:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792729043/real 1792729043] req@ffffa0a26aa8fa80 x1879812602171648/t0(0) o22->lustre-OST0000-osc-ffffa0a2141f2800@0@lo:6/4 lens 440/432 e 24 to 1 dl 1792729648 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0000-osc-ffffa0a2141f2800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-OST0000: Client 5d5ee6e0-2594-47d4-82cd-570e094068c8 (at 0@lo) reconnecting
Lustre: lustre-OST0000-osc-ffffa0a2141f2800: Connection restored to 0@lo (at 0@lo)
Lustre: 3048:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792729165/real 1792729165] req@ffffa0a28a71c600 x1879812619649024/t0(0) o22->lustre-OST0003-osc-ffffa0a2141f2800@0@lo:6/4 lens 440/432 e 4 to 1 dl 1792729770 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0003: Client 5d5ee6e0-2594-47d4-82cd-570e094068c8 (at 0@lo) reconnecting
Lustre: 59516:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffffa0a269d9db00 x1879812633453568/t0(0) o22->6687a0fe-0119-46ad-8fd8-b43e8afcd5ec@0@lo:481/0 lens 440/432 e 2 to 0 dl 1792729861 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 3051:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792729261/real 1792729261] req@ffffa0a284350380 x1879812633453568/t0(0) o22->lustre-OST0000-osc-ffffa0a255463800@0@lo:6/4 lens 440/432 e 2 to 1 dl 1792729866 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0000-osc-ffffa0a255463800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0000: Client 6687a0fe-0119-46ad-8fd8-b43e8afcd5ec (at 0@lo) reconnecting
Lustre: lustre-OST0000-osc-ffffa0a255463800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: mdt_out00_004: service thread pid 10062 was inactive for 42.564 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt00_007 state:I
Lustre: mdt00_001: service thread pid 5771 was inactive for 42.531 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
task:mdt_out00_004 state:I stack:0 pid:10062 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc27/0x1100 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2767/0x32f0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.50+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32f0/0x32f0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
stack:0 pid:8142 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
schedule+0xc4/0x180
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0xb7c/0x197e [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc27/0x1100 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x41b/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
mdt_create+0xa60/0x2140 [mdt]
? ldlm_request_cancel+0x6f7/0xba0 [ptlrpc]
mdt_reint_create+0x34d/0x5e0 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6ad/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 59516:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffffa0a2916ca680 x1879812636697088/t0(0) o22->6687a0fe-0119-46ad-8fd8-b43e8afcd5ec@0@lo:506/0 lens 440/432 e 2 to 0 dl 1792729886 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 3050:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792729286/real 1792729286] req@ffffa0a288d8f700 x1879812636697088/t0(0) o22->lustre-OST0003-osc-ffffa0a255463800@0@lo:6/4 lens 440/432 e 2 to 1 dl 1792729891 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0003-osc-ffffa0a255463800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: lustre-OST0003: Client 6687a0fe-0119-46ad-8fd8-b43e8afcd5ec (at 0@lo) reconnecting
Lustre: lustre-OST0003-osc-ffffa0a255463800: Connection restored to 0@lo (at 0@lo)
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: 00000000e4484751/0xcf90fd3c8bfa098b lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x40b5:0x0].0x0 bits 0x13/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xcf90fd3c8bfa093e expref: 1465 pid: 10062 timeout: 971 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: mdt00_012: service thread pid 8465 completed after 103.977s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_022: service thread pid 54846 completed after 104.016s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 10062:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 000000006c550aa2 ns: mdt-lustre-MDT0000_UUID lock: 00000000714a3527/0xcf90fd3c8bfa1864 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x3f52:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xcf90fd3c8bfa095a expref: 1435 pid: 10062 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: mdt00_001: service thread pid 5771 completed after 103.990s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_004: service thread pid 10062 completed after 104.023s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffffa0a2141f2800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffffa0a2141f2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 208634:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a2141f2800: inode [0x200000402:0x3d1e:0x0] mdc close failed: rc = -108
Lustre: mdt00_007: service thread pid 8142 completed after 104.029s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 39485 completed after 104.030s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 207837:0:(file.c:6392:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 207837:0:(file.c:6392:ll_inode_revalidate_fini()) Skipped 3 previous similar messages
Lustre: lustre-MDT0000-mdc-ffffa0a2141f2800: Connection restored to 0@lo (at 0@lo)
LustreError: 208582:0:(statahead.c:2479:start_statahead_thread()) lustre: cannot start ll_sa_208582 thread for 'racer': rc = -4
9[210784]: segfault at 0 ip 000055cc8330cb47 sp 00007ffffb2348e0 error 6 in 9[55cc83308000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 11979:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/4), not sending early reply req@ffffa0a29336b480 x1879812647289984/t0(0) o22->6687a0fe-0119-46ad-8fd8-b43e8afcd5ec@0@lo:566/0 lens 440/432 e 1 to 0 dl 1792729946 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: ll_ost00_000: service thread pid 6563 was inactive for 40.566 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
Lustre: ll_ost_io00_006: service thread pid 12518 was inactive for 633.542 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
4[222322]: segfault at 0 ip 000055ff46635b47 sp 00007ffd4c621370 error 6 in 4[55ff46631000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 40032:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (2/2), not sending early reply req@ffffa0a2814c0380 x1879812657803904/t0(0) o22->5d5ee6e0-2594-47d4-82cd-570e094068c8@0@lo:614/0 lens 440/432 e 1 to 0 dl 1792729994 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 000000005fd4b6d0/0xcf90fd3c8b7d9cc8 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0xbe:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 24576->36863) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0xcf90fd3c8b7d9cc1 expref: 17 pid: 6563 timeout: 1076 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: ll_ost00_000: service thread pid 6563 completed after 102.021s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_006: service thread pid 8821 completed after 102.023s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost00_011: service thread pid 26618 completed after 102.054s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_002: service thread pid 6574 completed after 993.964s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 5755:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792730037 with bad export cookie 14956732798976169185
Lustre: lustre-OST0000-osc-ffffa0a2141f2800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 6572:0:(service.c:2603:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (600/43s); client may timeout req@ffffa0a2814c0380 x1879812657803904/t0(0) o22->5d5ee6e0-2594-47d4-82cd-570e094068c8@0@lo:614/0 lens 440/432 e 1 to 0 dl 1792729994 ref 1 fl Complete:/600/0 rc -2/-2 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: ll_ost_io00_009: service thread pid 19825 completed after 872.104s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-OST0003-osc-ffffa0a2141f2800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-OST0000-osc-ffffa0a2141f2800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: lustre-OST0003-osc-ffffa0a2141f2800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 3044:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0002-osc-ffffa0a2141f2800: granted 3665920 but already consumed 4472832
Lustre: 3052:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0xa2a:0x0]/ may get corrupted (rc -108)
Lustre: 3053:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0xa43:0x0]// may get corrupted (rc -108)
LustreError: lustre-OST0002-osc-ffffa0a2141f2800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: lustre-OST0000-osc-ffffa0a2141f2800: Connection restored to 0@lo (at 0@lo)
Lustre: 3045:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0x9b1:0x0]/ may get corrupted (rc -108)
Lustre: 3048:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0x8ae:0x0]/ may get corrupted (rc -108)
Lustre: 3047:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0x98c:0x0]/ may get corrupted (rc -108)
Lustre: 3052:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0xa0d:0x0]/ may get corrupted (rc -108)
Lustre: 3054:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x4993:0x0]/ may get corrupted (rc -108)
Lustre: 3054:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000402:0x4947:0x0]/ may get corrupted (rc -108)
Lustre: 3056:0:(llite_lib.c:4335:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.97@tcp:/lustre/fid: [0x200000403:0xa45:0x0]// may get corrupted (rc -108)
LustreError: 239594:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffffa0a2141f2800: namespace resource [0x2c0000400:0x569:0x0].0x0 (ffffa0a279245b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 239594:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 3046:0:(client.c:2521:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792729346/real 1792729346] req@ffffa0a24b439f80 x1879812647289984/t0(0) o22->lustre-OST0001-osc-ffffa0a255463800@0@lo:6/4 lens 440/432 e 1 to 1 dl 1792729951 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0001: Client 6687a0fe-0119-46ad-8fd8-b43e8afcd5ec (at 0@lo) reconnecting
Lustre: ll_ost_out00_00: service thread pid 41529 was inactive for 42.864 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: 000000004f9fd1ea/0xcf90fd3c8c115c29 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x4a42:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xcf90fd3c8c115bdc expref: 347 pid: 41529 timeout: 1180 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
Lustre: mdt_io00_001: service thread pid 5784 completed after 100.221s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 41529:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 00000000823c9985 ns: mdt-lustre-MDT0000_UUID lock: 00000000775fb715/0xcf90fd3c8c1161d2 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x4a42:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xcf90fd3c8c115bf1 expref: 217 pid: 41529 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: ll_ost_out00_00: service thread pid 41529 completed after 100.248s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffffa0a2141f2800: operation mds_reint to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
LustreError: lustre-MDT0000-mdc-ffffa0a2141f2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 240062:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a2141f2800: inode [0x200000402:0x498c:0x0] mdc close failed: rc = -108
LustreError: 240062:0:(file.c:252:ll_close_inode_openhandle()) Skipped 5 previous similar messages
LustreError: 240062:0:(llite_lib.c:2150:ll_md_setattr()) md_setattr fails: rc = -108
LustreError: 240230:0:(file.c:6392:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 240230:0:(file.c:6392:ll_inode_revalidate_fini()) Skipped 220 previous similar messages
LustreError: 240212:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0a2141f2800: namespace resource [0x200000007:0x1:0x0].0x0 (ffffa0a2046e8700) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 240212:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 47164:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 609, rollback = 7
Lustre: 47164:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 266 previous similar messages
Lustre: 47164:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 47164:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 267 previous similar messages
Lustre: 47164:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 47164:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 267 previous similar messages
Lustre: 47164:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/609/0, punch: 0/0/0, quota 1/3/0
Lustre: 47164:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 267 previous similar messages
Lustre: 47164:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 47164:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 267 previous similar messages
Lustre: 47164:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 47164:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 267 previous similar messages
LustreError: 5759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: 0000000003d6886a/0xcf90fd3c8c11cdf0 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x486a:0x0].0x0 bits 0x13/0x0 rrc: 9 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xcf90fd3c8c11cd9c expref: 78 pid: 10062 timeout: 1282 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 17589:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 000000005a493496 ns: mdt-lustre-MDT0000_UUID lock: 00000000fc6fd9bf/0xcf90fd3c8c11d019 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x486a:0x0].0x0 bits 0x13/0x0 rrc: 10 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xcf90fd3c8c11d00b expref: 74 pid: 17589 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-MDT0000-mdc-ffffa0a2141f2800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffffa0a2141f2800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 5 previous similar messages
Lustre: Skipped 4 previous similar messages
LustreError: lustre-MDT0000-mdc-ffffa0a2141f2800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 240778:0:(file.c:6392:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
LustreError: 240778:0:(file.c:6392:ll_inode_revalidate_fini()) Skipped 44 previous similar messages
LustreError: 240823:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a2141f2800: inode [0x200000403:0xa3f:0x0] mdc close failed: rc = -108
LustreError: 240823:0:(file.c:252:ll_close_inode_openhandle()) Skipped 6 previous similar messages
LustreError: 240823:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0000-mdc-ffffa0a2141f2800: namespace resource [0x200000401:0x1:0x0].0x0 (ffffa0a2824c8b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-MDT0000-mdc-ffffa0a2141f2800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 4 previous similar messages
Lustre: 38025:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
19[259295]: segfault at 8 ip 00007feae5fe6875 sp 00007ffe0da8dfa0 error 4 in ld-2.28.so[7feae5fc5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[264831]: segfault at 0 ip 000055901eb8fb47 sp 00007ffe2075cd40 error 6 in 15[55901eb8b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3[265040]: segfault at 0 ip 00005631f438cb47 sp 00007ffdf2e80570 error 6 in 3[5631f4388000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 276125:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa0a2141f2800: inode [0x200000405:0xbd3:0x0] mdc close failed: rc = -13
LustreError: 276125:0:(file.c:252:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 289830:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
1[294563]: segfault at 8 ip 00007fa480084875 sp 00007ffe00638130 error 4 in ld-2.28.so[7fa480063000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48

LustreError: 49900:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 49900:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 14 PID: 49900 Comm: ll_sa_49281 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 7881:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff897e2a27ec80 x1879718687883776/t4294968141(0) o101->b0608e84-77be-4fc4-b170-fda026092815@0@lo:702/0 lens 376/864 e 0 to 0 dl 1792639482 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 10912:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff897e5922f800: inode [0x200000402:0x9:0x0] mdc close failed: rc = -13
15[12275]: segfault at 8 ip 00007fef59aed875 sp 00007ffc2532a850 error 4 in ld-2.28.so[7fef59acc000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11629:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 11629:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11629:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 11629:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11629:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11629:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8199:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8199:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8199:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 8199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8199:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8199:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8199:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8199:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8199:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10998:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 10998:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10998:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10998:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10998:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10998:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10998:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 10998:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10998:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10998:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10998:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10998:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 24277:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff897e5922f800: inode [0x200000401:0x535:0x0] mdc close failed: rc = -13
Lustre: ll_ost_io00_002: service thread pid 6523 was inactive for 40.482 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_002 state:D stack:0 pid:6523 ppid:2 flags:0x80004080
task:ll_ost_io00_000 state:R running task stack:0 pid:6520 ppid:2 flags:0x80004080
Call Trace:
Call Trace:
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
__schedule+0x351/0xcb0
? __ldiskfs_journal_get_write_access+0x64/0xd0 [ldiskfs]
? osd_ldiskfs_write+0x77b/0x830 [osd_ldiskfs]
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
? __schedule+0x351/0xcb0
? schedule+0xc4/0x180
? wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
? add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
? jbd2__journal_start+0x11f/0x300
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? dt_trans_start+0x20/0x70 [obdclass]
? ofd_trans_start+0x7d/0xf0 [ofd]
? ofd_object_fallocate+0x6cf/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 11629:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 11629:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 11629:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11629:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11629:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 11629:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11629:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11629:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11629:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11629:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11629:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11629:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 23905:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 23905:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 23905:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 23905:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 23905:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 23905:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 23905:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 23905:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 23905:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 23905:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 23905:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 23905:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8199:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8199:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 8199:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8199:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 8199:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8199:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 8199:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8199:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 8199:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8199:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 8199:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8199:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: ll_ost_io00_001: service thread pid 6522 was inactive for 73.591 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
task:ll_ost_io00_001 state:R running task stack:0 pid:6522 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
mutex_lock+0x35/0x80
tgt_last_rcvd_update+0x2b5/0xec0 [ptlrpc]
tgt_txn_stop_cb+0x123/0x680 [ptlrpc]
? jbd2_journal_extend+0x7d/0x2b0
dt_txn_hook_stop+0xf1/0x1d0 [obdclass]
osd_trans_stop+0x43e/0xe10 [osd_ldiskfs]
ofd_trans_stop+0x32/0x90 [ofd]
ofd_object_fallocate+0x99a/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_io00_007: service thread pid 12004 was inactive for 75.533 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
13[46042]: segfault at 0 ip 0000562e7cbb0b47 sp 00007ffc865f28f0 error 6 in 13[562e7cbac000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 5755:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
LustreError: 5731:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000afd26987/0x62928a197dcfba89 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x17:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 24576->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62928a197dcfba7b expref: 72 pid: 8490 timeout: 171 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff897e13d50800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff897e13d50800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff897e13d50800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3003:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000401:0xda5:0x0]// may get corrupted (rc -108)
LustreError: 48302:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff897e13d50800: namespace resource [0x300000400:0x17:0x0].0x0 (ffff897dffc4f700) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff897e13d50800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_000: service thread pid 6520 completed after 103.852s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_002: service thread pid 6523 completed after 102.795s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 5754:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5754:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 5731:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000001420b213/0x62928a197dd4622d lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x40:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->12287) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62928a197dd4620a expref: 9399 pid: 9330 timeout: 187 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffff897e5922f800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff897e5922f800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff897e5922f800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3006:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000401:0xe63:0x0]/ may get corrupted (rc -108)
Lustre: 3006:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000402:0xef8:0x0]/ may get corrupted (rc -108)
Lustre: 2998:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000401:0xe80:0x0]// may get corrupted (rc -108)
Lustre: 2995:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000402:0xf1c:0x0]/ may get corrupted (rc -108)
Lustre: 3000:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.7@tcp:/lustre/fid: [0x200000402:0xfe3:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff897e5922f800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_001: service thread pid 6522 completed after 101.703s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 10998:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10998:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 21 previous similar messages
Lustre: 10998:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10998:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 24 previous similar messages
Lustre: 10998:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10998:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 24 previous similar messages
Lustre: 10998:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10998:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 24 previous similar messages
Lustre: 10998:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10998:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 24 previous similar messages
Lustre: 10998:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10998:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 24 previous similar messages

LustreError: 140274:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 140274:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 10 PID: 140274 Comm: ll_sa_139917 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 8404:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff95e10955ad80 x1879542179921792/t4294967926(0) o101->afa657f1-f358-4e52-9485-5a36832daf0e@0@lo:737/0 lens 376/864 e 0 to 0 dl 1792471152 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
8[14047]: segfault at 8 ip 00007f9eee3da875 sp 00007ffd83ffd0d0 error 4 in ld-2.28.so[7f9eee3b9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12954:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 12954:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12954:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 12954:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 12954:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12954:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12954:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7922:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7922:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7922:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7922:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7922:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7922:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7922:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7922:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 7922:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7922:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7922:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7922:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7922:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7922:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7922:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: ll_ost_io00_002: service thread pid 6321 was inactive for 42.765 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_002 state:R running task stack:0 pid:6321 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
preempt_schedule_common+0x2f/0x60
? osd_declare_qid+0x40b/0x7c0 [osd_ldiskfs]
? osd_fallocate_preallocate.isra.36+0x396/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 12954:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 12954:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12954:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 12954:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 12954:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12954:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12954:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12954:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: ll_ost_io00_006: service thread pid 15775 was inactive for 73.945 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_006 state:D stack:0 pid:15775 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 11 previous similar messages
LustreError: 5526:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000331b4939/0xb38aa81779f9d2ae lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x27:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 8192->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xb38aa81779f9d27d expref: 360 pid: 9094 timeout: 207 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff95e120343000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff95e120343000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 12954:0:(tgt_grant.c:803:tgt_grant_check()) lustre-OST0001: cli afa657f1-f358-4e52-9485-5a36832daf0e claims 1466368 GRANT, real grant 0
LustreError: 12954:0:(ldlm_lib.c:3676:target_bulk_io()) @@@ bulk WRITE failed: rc = -107 req@ffff95e110665780 x1879542201434240/t0(0) o4->afa657f1-f358-4e52-9485-5a36832daf0e@0@lo:276/0 lens 488/448 e 0 to 0 dl 1792471446 ref 1 fl Interpret:H/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0001: Bulk IO write error with afa657f1-f358-4e52-9485-5a36832daf0e (at 0@lo), client will retry: rc = -107
LustreError: lustre-OST0001-osc-ffff95e120343000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3046:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x976:0x0]// may get corrupted (rc -5)
Lustre: 3053:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x94b:0x0]/ may get corrupted (rc -108)
Lustre: 3052:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x9e0:0x0]/ may get corrupted (rc -108)
LustreError: 38772:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff95e120343000: namespace resource [0x280000400:0x27:0x0].0x0 (ffff95e0d3251c00) refcount nonzero (2) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff95e120343000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_002: service thread pid 6321 completed after 118.682s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_006: service thread pid 15775 completed after 106.249s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 43 previous similar messages
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10523:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10523:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10523:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10523:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10523:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 43 previous similar messages
Lustre: 9060:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
INFO: task fallocate:39673 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:39673 ppid:7527 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f62e6a2b70b
Code: Unable to access opcode bytes at RIP 0x7f62e6a2b6e1.
RSP: 002b:00007ffc3779c4d8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f62e6a2b70b
RDX: 00000000000018b4 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006e49 R11: 0000000000000246 R12: 00007ffc3779c6f8
R13: 0000000000000001 R14: 00007ffc3779c538 R15: 00000000000018b4
INFO: task fallocate:39734 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:39734 ppid:7502 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
vvp_io_setattr_start+0x573/0x5f0 [lustre]
? init_wait_var_entry+0x50/0x50
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f40aba4270b
Code: Unable to access opcode bytes at RIP 0x7f40aba426e1.
RSP: 002b:00007ffefd6b7578 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f40aba4270b
RDX: 0000000000007539 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004039 R11: 0000000000000246 R12: 00007ffefd6b7798
R13: 0000000000000000 R14: 00007ffefd6b75d8 R15: 0000000000007539
LustreError: 5526:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000fc752a06/0xb38aa8177a13ccae lrc: 3/0,0 mode: PW/PW res: [0x300000400:0xff:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 4096->36863) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xb38aa8177a13cca7 expref: 62346 pid: 8076 timeout: 375 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff95e118928800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff95e118928800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff95e118928800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3050:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x1281:0x0]// may get corrupted (rc -108)
Lustre: 3050:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x11bc:0x0]/ may get corrupted (rc -108)
LustreError: 67806:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff95e118928800: namespace resource [0x300000400:0x14c:0x0].0x0 (ffff95e140b99500) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 67806:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-ffff95e118928800: Connection restored to 0@lo (at 0@lo)
Lustre: 6320:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6320:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 73 previous similar messages
Lustre: 6320:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6320:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 6320:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6320:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 6320:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6320:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 6320:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6320:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 6320:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6320:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 5549:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5549:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 34428:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 34428:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
INFO: task fallocate:39673 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
LustreError: 5526:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000216b37ca/0xb38aa8177a22b616 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x172:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xb38aa8177a22b60f expref: 11215 pid: 15765 timeout: 490 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:39673 ppid:7527 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f62e6a2b70b
Code: Unable to access opcode bytes at RIP 0x7f62e6a2b6e1.
RSP: 002b:00007ffc3779c4d8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f62e6a2b70b
RDX: 00000000000018b4 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006e49 R11: 0000000000000246 R12: 00007ffc3779c6f8
R13: 0000000000000001 R14: 00007ffc3779c538 R15: 00000000000018b4
LustreError: lustre-OST0002-osc-ffff95e120343000: operation ldlm_enqueue to node 0@lo failed: rc = -107
INFO: task fallocate:39734 blocked for more than 120 seconds.
Lustre: lustre-OST0002-osc-ffff95e120343000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
LustreError: lustre-OST0002-osc-ffff95e120343000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:39734 ppid:7502 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
vvp_io_setattr_start+0x573/0x5f0 [lustre]
? init_wait_var_entry+0x50/0x50
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f40aba4270b
Code: Unable to access opcode bytes at RIP 0x7f40aba426e1.
RSP: 002b:00007ffefd6b7578 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f40aba4270b
RDX: 0000000000007539 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004039 R11: 0000000000000246 R12: 00007ffefd6b7798
R13: 0000000000000000 R14: 00007ffefd6b75d8 R15: 0000000000007539
INFO: task fallocate:49865 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:49865 ppid:7479 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fcd6772f70b
Code: Unable to access opcode bytes at RIP 0x7fcd6772f6e1.
RSP: 002b:00007ffe89ccbec8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcd6772f70b
RDX: 0000000000007b0b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000257f R11: 0000000000000246 R12: 00007ffe89ccc0e8
Lustre: 3038:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x1b0e:0x0]// may get corrupted (rc -108)
R13: 0000000000000000 R14: 00007ffe89ccbf28 R15: 0000000000007b0b
INFO: task chown:49973 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:49973 ppid:7489 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5d0578efbe
Code: Unable to access opcode bytes at RIP 0x7f5d0578ef94.
RSP: 002b:00007ffd24b46e18 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffd24b47050 RCX: 00007f5d0578efbe
RDX: 0000000000003225 RSI: 00005629400f0d20 RDI: 00000000ffffff9c
RBP: 00005629400f1f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000006e5a R11: 0000000000000246 R12: 00000000ffffffff
R13: 00005629400f1fc8 R14: 0000000000003225 R15: 0000000000006e5a
Lustre: 3040:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x1a06:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff95e120343000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_006: service thread pid 15775 was inactive for 280.212 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_006 state:R running task stack:0 pid:15775 ppid:2 flags:0x80004080
Call Trace:
? slab_free_freelist_hook.isra.49+0x111/0x230
? osd_trunc_unlock_all+0xf2/0x1e0 [osd_ldiskfs]
? kfree+0x28a/0x380
? osd_trans_stop+0x5cf/0xe10 [osd_ldiskfs]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x99a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
10[90667]: segfault at 8 ip 00007f871606c875 sp 00007ffd4c793f70 error 4 in ld-2.28.so[7f871604b000+2f000]
10[90812]: segfault at 8 ip 00007f5261acc875 sp 00007ffda77b4650 error 4 in ld-2.28.so[7f5261aab000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 91253:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff95e118928800: inode [0x200000401:0x1b1f:0x0] mdc close failed: rc = -13
LustreError: 99026:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff95e118928800: inode [0x200000402:0x1e1c:0x0] mdc close failed: rc = -13
Lustre: ll_ost_io00_003: service thread pid 7922 was inactive for 292.539 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_003 state:D stack:0 pid:7922 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? bit_wait+0x90/0x90
? set_next_entity+0xb4/0xd0
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock_irq+0x16/0x50
? __find_get_block+0x118/0x540
? ldiskfs_es_lookup_extent+0xd7/0x350 [ldiskfs]
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
? __getblk_gfp+0x59/0x5b0
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? __ldiskfs_journal_get_write_access+0x64/0xd0 [ldiskfs]
? osd_ldiskfs_write+0x77b/0x830 [osd_ldiskfs]
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? slab_free_freelist_hook.isra.49+0x111/0x230
? osd_trunc_unlock_all+0xf2/0x1e0 [osd_ldiskfs]
? kfree+0x28a/0x380
? __check_object_size+0x10e/0x26d
? __probe_kernel_read+0x82/0x100
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? kmem_cache_alloc_trace+0x215/0x440
? __might_sleep+0x60/0xc0
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? ofd_trans_create+0x7f/0x2b0 [ofd]
? ofd_object_fallocate+0x36b/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
3[104574]: segfault at 0 ip 000055ef5e179b47 sp 00007ffe574fee50 error 6 in 3[55ef5e175000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 5526:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000002de6e21d/0xb38aa8177a344f06 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x20e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 24576->61439) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xb38aa8177a344ee3 expref: 723 pid: 6480 timeout: 608 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff95e120343000: operation ost_punch to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff95e120343000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff95e120343000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3043:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x2377:0x0]// may get corrupted (rc -108)
Lustre: 3047:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x23b3:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff95e120343000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_006: service thread pid 15775 completed after 396.892s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
INFO: task fallocate:49865 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:49865 ppid:7479 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fcd6772f70b
Code: Unable to access opcode bytes at RIP 0x7fcd6772f6e1.
RSP: 002b:00007ffe89ccbec8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcd6772f70b
RDX: 0000000000007b0b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000257f R11: 0000000000000246 R12: 00007ffe89ccc0e8
R13: 0000000000000000 R14: 00007ffe89ccbf28 R15: 0000000000007b0b
INFO: task chown:49973 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:49973 ppid:7489 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5d0578efbe
Code: Unable to access opcode bytes at RIP 0x7f5d0578ef94.
RSP: 002b:00007ffd24b46e18 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffd24b47050 RCX: 00007f5d0578efbe
RDX: 0000000000003225 RSI: 00005629400f0d20 RDI: 00000000ffffff9c
RBP: 00005629400f1f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000006e5a R11: 0000000000000246 R12: 00000000ffffffff
R13: 00005629400f1fc8 R14: 0000000000003225 R15: 0000000000006e5a
15[116439]: segfault at 55f2e7947000 ip 000055f2e7947000 sp 00007fff51a193e8 error 14 in 15[55f2e7b47000+1000]
Code: Unable to access opcode bytes at RIP 0x55f2e7946fd6.
15[116370]: segfault at 558a220f3000 ip 0000558a220f3000 sp 00007ffd396e6038 error 14 in 15[558a222f3000+1000]
Code: Unable to access opcode bytes at RIP 0x558a220f2fd6.
Lustre: 9060:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 9060:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 9060:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 9060:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 9060:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0
Lustre: 9060:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 9060:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 7/249/0
Lustre: 9060:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 9060:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 7/117/1, delete: 3/6/1
Lustre: 9060:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 9060:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 9060:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10523:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 123 previous similar messages
traps: 2[130702] trap invalid opcode ip:557e13f2a66c sp:7ffc5578ae80 error:0 in 6[557e13f25000+7000]
LustreError: 5526:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000d9a1c2b0/0xb38aa8177a457231 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x2ae:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 24576->53247) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xb38aa8177a45722a expref: 18 pid: 8917 timeout: 715 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff95e120343000: operation ost_read to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff95e120343000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff95e120343000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3053:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x2b6c:0x0]/ may get corrupted (rc -108)
Lustre: 3038:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x2b56:0x0]/ may get corrupted (rc -108)
LustreError: 134321:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff95e120343000: namespace resource [0x2c0000400:0x31a:0x0].0x0 (ffff95e111cb3700) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 134321:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff95e120343000: Connection restored to 0@lo (at 0@lo)
INFO: task fallocate:49865 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:49865 ppid:7479 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fcd6772f70b
Code: Unable to access opcode bytes at RIP 0x7fcd6772f6e1.
RSP: 002b:00007ffe89ccbec8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fcd6772f70b
RDX: 0000000000007b0b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000257f R11: 0000000000000246 R12: 00007ffe89ccc0e8
R13: 0000000000000000 R14: 00007ffe89ccbf28 R15: 0000000000007b0b
INFO: task chown:49973 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:49973 ppid:7489 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5d0578efbe
Code: Unable to access opcode bytes at RIP 0x7f5d0578ef94.
RSP: 002b:00007ffd24b46e18 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffd24b47050 RCX: 00007f5d0578efbe
RDX: 0000000000003225 RSI: 00005629400f0d20 RDI: 00000000ffffff9c
RBP: 00005629400f1f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000006e5a R11: 0000000000000246 R12: 00000000ffffffff
R13: 00005629400f1fc8 R14: 0000000000003225 R15: 0000000000006e5a
LustreError: 138243:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff95e120343000: inode [0x200000402:0x2b08:0x0] mdc close failed: rc = -13

LustreError: 527557:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 527557:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 12 PID: 527557 Comm: ll_sa_525815 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 7744:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7744:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7744:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7744:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7744:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7744:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
8[10602]: segfault at 8 ip 00007f25bfc2d875 sp 00007fffe844f340 error 4 in ld-2.28.so[7f25bfc0c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[11308]: segfault at 8 ip 00007f01ed3f0875 sp 00007ffc236918a0 error 4 in ld-2.28.so[7f01ed3cf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 10721:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 10721:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10721:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10721:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10721:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10721:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10721:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10721:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10721:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10721:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10721:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10721:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10721:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10721:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 10721:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10721:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10721:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10721:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10721:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10721:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10721:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10721:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10721:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10721:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 13597:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff980b89432a00 x1879314585392896/t4294968632(0) o101->2dfe7373-8d79-477f-9fa5-b8053ca841db@0@lo:408/0 lens 376/840 e 0 to 0 dl 1792254138 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 7744:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7744:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 7744:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7744:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7744:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7744:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7744:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7744:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7744:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7744:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7744:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7744:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9714:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff980bf6b77a80 x1879314585776128/t4294968970(0) o101->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:412/0 lens 376/816 e 0 to 0 dl 1792254142 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
18[15593]: segfault at 5569a9b78000 ip 00005569a9b78000 sp 00007fff11c964d8 error 14 in 18[5569a9d78000+1000]
Code: Unable to access opcode bytes at RIP 0x5569a9b77fd6.
systemd-coredump[16108]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[16108]: Failed to get EXE, ignoring: No such process
Lustre: 10917:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10917:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10917:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10917:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10917:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10917:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10917:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10917:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10917:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10917:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10917:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10917:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: ll_ost_io00_004: service thread pid 10514 was inactive for 40.449 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_004 state:R running task stack:0 pid:10514 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 16707:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff980bf552c600 x1879314589278720/t4294969644(0) o101->2dfe7373-8d79-477f-9fa5-b8053ca841db@0@lo:449/0 lens 376/864 e 0 to 0 dl 1792254179 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 9972:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 9972:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 9972:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9972:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9972:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 9972:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9972:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 9972:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9972:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9972:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 9972:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9972:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 20441:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 20441:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 20441:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20441:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 20441:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 20441:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 20441:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 20441:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 20441:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20441:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 20441:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20441:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: ll_ost_io00_007: service thread pid 13118 was inactive for 75.443 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_007 state:R running task stack:0 pid:13118 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
slab_pre_alloc_hook.constprop.64+0x138/0x1f0
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
kmem_cache_alloc_trace+0x5b/0x440
? osd_declare_inode_qid+0x48f/0x6c0 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? dt_trans_start+0x20/0x70 [obdclass]
? ofd_trans_start+0x7d/0xf0 [ofd]
? ofd_object_fallocate+0x6cf/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
17[27207]: segfault at 8 ip 00007f3afe1e2875 sp 00007fffc6c93b60 error 4 in ld-2.28.so[7f3afe1c1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6426:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000f2b52261/0x3bd7133cab58a723 lrc: 3/0,0 mode: PW/PW res: [0x2c0000402:0xb:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 20480->32767) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cab58a6e4 expref: 321 pid: 7730 timeout: 217 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff980c0792e000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff980c0792e000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff980c0792e000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x109:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x1da:0x0]// may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x14b:0x0]// may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x101:0x0]/ may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x1dc:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x1cf:0x0]// may get corrupted (rc -108)
LustreError: 28154:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff980c0792e000: namespace resource [0x2c0000402:0x17:0x0].0x0 (ffff980b84f9fd00) refcount nonzero (2) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff980c0792e000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_006: service thread pid 10917 was inactive for 74.920 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_006 state:R running task stack:0 pid:10917 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
down+0x39/0xa0
osd_ldiskfs_write+0x2f4/0x830 [osd_ldiskfs]
osd_write+0x34d/0x650 [osd_ldiskfs]
dt_record_write+0x3f/0x190 [obdclass]
tgt_client_data_write.isra.13+0x1a6/0x210 [ptlrpc]
tgt_last_rcvd_update+0x39f/0xec0 [ptlrpc]
tgt_txn_stop_cb+0x123/0x680 [ptlrpc]
? jbd2_journal_extend+0x7d/0x2b0
dt_txn_hook_stop+0xf1/0x1d0 [obdclass]
osd_trans_stop+0x43e/0xe10 [osd_ldiskfs]
ofd_trans_stop+0x32/0x90 [ofd]
ofd_object_fallocate+0x99a/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_io00_004: service thread pid 10514 completed after 108.161s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 20441:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 20441:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 20441:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 20441:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 20441:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 20441:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 20441:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 20441:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 20441:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 20441:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 20441:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 20441:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 11 previous similar messages
17[35884]: segfault at 8 ip 00007f835ba6c875 sp 00007ffe01c0a5a0 error 4 in ld-2.28.so[7f835ba4b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[36350]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[36350]: Failed to get EXE, ignoring: No such process
Lustre: 24230:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 24230:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 24230:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 24230:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 24230:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 24230:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 24230:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 24230:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 24230:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 24230:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 24230:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 24230:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 11 previous similar messages
15[44761]: segfault at 8 ip 00007fcca3bd0875 sp 00007fff046e70a0 error 4 in ld-2.28.so[7fcca3baf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 108s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000004703c74a/0x3bd7133cab63378b lrc: 3/0,0 mode: PW/PW res: [0x2c0000402:0x17:0x0].0x0 rrc: 7 type: EXT [16384->20479] (req 16384->20479) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x3bd7133cab6336ff expref: 6210 pid: 8251 timeout: 341 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
systemd-coredump[45362]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[45362]: Failed to get EXE, ignoring: No such process
LustreError: lustre-OST0001-osc-ffff980c05ae3800: operation ost_read to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff980c05ae3800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff980c0792e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
LustreError: lustre-OST0001-osc-ffff980c05ae3800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x4cb:0x0]// may get corrupted (rc -108)
LustreError: lustre-OST0002-osc-ffff980c0792e000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x4b9:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x2dd:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x37e:0x0]// may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x367:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x462:0x0]/ may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x416:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4ad:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2a4:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x48b:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff980c05ae3800: Connection restored to 0@lo (at 0@lo)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x347:0x0]// may get corrupted (rc -5)
LustreError: lustre-OST0003-osc-ffff980c0792e000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x4e2:0x0]// may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2ac:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x387:0x0]// may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x276:0x0]/ may get corrupted (rc -5)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x347:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x359:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x376:0x0]/ may get corrupted (rc -108)
Lustre: ll_ost_io00_007: service thread pid 13118 completed after 226.449s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-OST0003-osc-ffff980c0792e000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 2 previous similar messages
Lustre: ll_ost_io00_006: service thread pid 10917 completed after 208.714s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
INFO: task fallocate:28763 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:28763 ppid:9123 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fda8bee870b
Code: Unable to access opcode bytes at RIP 0x7fda8bee86e1.
RSP: 002b:00007ffe19f8a398 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fda8bee870b
RDX: 000000000000335b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000059cf R11: 0000000000000246 R12: 00007ffe19f8a5b8
R13: 0000000000000000 R14: 00007ffe19f8a3f8 R15: 000000000000335b
14[59572]: segfault at 55baa8d24000 ip 000055baa8d24000 sp 00007ffc9f4bf788 error 14 in 14[55baa8f24000+1000]
Code: Unable to access opcode bytes at RIP 0x55baa8d23fd6.
systemd-coredump[60176]: Failed to get EXE, ignoring: No such process
Lustre: 7744:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 7744:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 37 previous similar messages
Lustre: 7744:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7744:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 7744:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7744:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 7744:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7744:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 7744:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7744:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 37 previous similar messages
Lustre: 7744:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7744:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 37 previous similar messages
LustreError: 63258:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x280000403:0x426:0x0] mdc close failed: rc = -13
0[63900]: segfault at 4 ip 00005566c3efa3a0 sp 00007fff5e169d68 error 6 in 0[5566c3ef8000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
systemd-coredump[64430]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[64430]: Failed to get EXE, ignoring: No such process
15[65883]: segfault at 8 ip 00007f6617160875 sp 00007ffc324ebc80 error 4 in ld-2.28.so[7f661713f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:28763 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:28763 ppid:9123 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fda8bee870b
Code: Unable to access opcode bytes at RIP 0x7fda8bee86e1.
RSP: 002b:00007ffe19f8a398 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fda8bee870b
RDX: 000000000000335b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000059cf R11: 0000000000000246 R12: 00007ffe19f8a5b8
R13: 0000000000000000 R14: 00007ffe19f8a3f8 R15: 000000000000335b
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000003af08634/0x3bd7133cab771486 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x68:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 28672->61439) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cab77146a expref: 596 pid: 31650 timeout: 514 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 3 previous similar messages
LustreError: lustre-OST0000-osc-ffff980c05ae3800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 4 previous similar messages
Lustre: lustre-OST0000-osc-ffff980c05ae3800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
LustreError: lustre-OST0000-osc-ffff980c05ae3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x492:0x0]// may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x6a1:0x0]// may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x4de:0x0]// may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x60f:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x635:0x0]/ may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x541:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff980c05ae3800: Connection restored to 0@lo (at 0@lo)
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000e78ce1f4/0x3bd7133cab7b2f3c lrc: 3/0,0 mode: PW/PW res: [0x340000402:0xaa:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 16384->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cab7b2e86 expref: 191 pid: 23556 timeout: 547 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 7733:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff980bf6669c00 x1879314636572544/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff980c05ae3800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff980c05ae3800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff980c05ae3800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x84a:0x0]// may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x6ce:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x652:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x515:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x602:0x0]/ may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x684:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x768:0x0]// may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x8bd:0x0]// may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x8ed:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff980c05ae3800: Connection restored to 0@lo (at 0@lo)
5[82438]: segfault at 8 ip 00007f34782ef875 sp 00007ffdc34159c0 error 4 in ld-2.28.so[7f34782ce000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000070a3d0d6/0x3bd7133cab8125dd lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x8f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 20480->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cab812551 expref: 26 pid: 15719 timeout: 588 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff980c05ae3800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff980c05ae3800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff980c05ae3800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x6c9:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x998:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x719:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0xa99:0x0]// may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x72b:0x0]// may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0xa7a:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff980c05ae3800: Connection restored to 0@lo (at 0@lo)
INFO: task fallocate:52531 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:52531 ppid:9273 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5a1668d70b
Code: Unable to access opcode bytes at RIP 0x7f5a1668d6e1.
RSP: 002b:00007ffd5fc8fc78 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5a1668d70b
RDX: 000000000000797a RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006e89 R11: 0000000000000246 R12: 00007ffd5fc8fe98
R13: 0000000000000001 R14: 00007ffd5fc8fcd8 R15: 000000000000797a
INFO: task fallocate:58829 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:58829 ppid:9359 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fed462bd70b
Code: Unable to access opcode bytes at RIP 0x7fed462bd6e1.
RSP: 002b:00007ffcef0aa948 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fed462bd70b
RDX: 000000000000419f RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003f17 R11: 0000000000000246 R12: 00007ffcef0aab68
R13: 0000000000000000 R14: 00007ffcef0aa9a8 R15: 000000000000419f
3[101148]: segfault at 8 ip 00007f89cc515875 sp 00007fffa2883710 error 4 in ld-2.28.so[7f89cc4f4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[101814]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[101814]: Failed to get EXE, ignoring: No such process
1[102767]: segfault at 5646ca717000 ip 00005646ca717000 sp 00007ffca1fee0e0 error 14 in 1[5646ca917000+1000]
Code: Unable to access opcode bytes at RIP 0x5646ca716fd6.
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000befee249/0x3bd7133cab91292f lrc: 3/0,0 mode: PW/PW res: [0x380000400:0xcf:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->36863) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cab912913 expref: 24878 pid: 8251 timeout: 688 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff980c0792e000: operation ost_punch to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff980c0792e000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff980c0792e000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0xa7c:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x968:0x0]/ may get corrupted (rc -108)
LustreError: 107773:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff980c0792e000: namespace resource [0x380000402:0x13f:0x0].0x0 (ffff980c0e372600) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0xd15:0x0]// may get corrupted (rc -108)
LustreError: 107773:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff980c0792e000: Connection restored to 0@lo (at 0@lo)
LustreError: 109848:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c0792e000: inode [0x200000404:0xd52:0x0] mdc close failed: rc = -13
Lustre: 13118:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 591, rollback = 7
Lustre: 13118:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 145 previous similar messages
Lustre: 13118:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 13118:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 13118:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 13118:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 13118:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/591/0, punch: 0/0/0, quota 4/150/0
Lustre: 13118:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 13118:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 13118:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 145 previous similar messages
Lustre: 13118:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 13118:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 145 previous similar messages
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000008050f0e9/0x3bd7133cab94f19e lrc: 3/0,0 mode: PW/PW res: [0x300000402:0x112:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cab94f197 expref: 871 pid: 8251 timeout: 712 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffff980c05ae3800: operation ost_punch to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff980c05ae3800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff980c05ae3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x9e1:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x97a:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0xdd2:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0xe04:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0xba9:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0xc04:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x9b9:0x0]// may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0xdc0:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0xb75:0x0]// may get corrupted (rc -108)
Lustre: 13341:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 13341:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
INFO: task fallocate:52531 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:52531 ppid:9273 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5a1668d70b
Code: Unable to access opcode bytes at RIP 0x7f5a1668d6e1.
RSP: 002b:00007ffd5fc8fc78 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5a1668d70b
RDX: 000000000000797a RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006e89 R11: 0000000000000246 R12: 00007ffd5fc8fe98
R13: 0000000000000001 R14: 00007ffd5fc8fcd8 R15: 000000000000797a
INFO: task fallocate:58829 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:58829 ppid:9359 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fed462bd70b
Code: Unable to access opcode bytes at RIP 0x7fed462bd6e1.
RSP: 002b:00007ffcef0aa948 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fed462bd70b
RDX: 000000000000419f RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003f17 R11: 0000000000000246 R12: 00007ffcef0aab68
R13: 0000000000000000 R14: 00007ffcef0aa9a8 R15: 000000000000419f
INFO: task fallocate:67653 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:67653 ppid:9376 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7effb180770b
Code: Unable to access opcode bytes at RIP 0x7effb18076e1.
RSP: 002b:00007fff1d191a68 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007effb180770b
RDX: 0000000000004ff5 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000943 R11: 0000000000000246 R12: 00007fff1d191c88
R13: 0000000000000000 R14: 00007fff1d191ac8 R15: 0000000000004ff5
INFO: task touch:82754 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:touch state:D stack:0 pid:82754 ppid:9375 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? dput+0x1b7/0x310
? mntput+0x2c/0x60
? terminate_walk+0x141/0x160
? security_capable+0x41/0xb0
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
utimes_common.isra.2+0x14c/0x2f0
do_utimes+0x1f2/0x280
__x64_sys_utimensat+0xa6/0x130
? fput+0x1b/0x25
? __audit_syscall_exit+0xe6/0x160
? filp_close+0x77/0xa0
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f6bc1a592f7
Code: Unable to access opcode bytes at RIP 0x7f6bc1a592cd.
RSP: 002b:00007fff1b4e38e8 EFLAGS: 00000246 ORIG_RAX: 0000000000000118
RAX: ffffffffffffffda RBX: 00007fff1b4e5b4e RCX: 00007f6bc1a592f7
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000000
RBP: 0000000000000000 R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000ffffff9c
R13: 0000000000000000 R14: 00007fff1b4e5b4e R15: 00007f6bc1cf7374
LustreError: 123146:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x240000404:0xce5:0x0] mdc close failed: rc = -13
traps: 10[126923] general protection fault ip:562c0e6e7dac sp:7ffededb6f08 error:0 in 10[562c0e6e1000+7000]
0[126249]: segfault at 8 ip 00007f3bae1b2875 sp 00007ffcdbd17570 error 4 in ld-2.28.so[7f3bae191000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[127127]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[127127]: Failed to get EXE, ignoring: No such process
16[139094]: segfault at 8 ip 00007fb1ae12b875 sp 00007fff65b0a590 error 4 in ld-2.28.so[7fb1ae10a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:52531 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:52531 ppid:9273 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5a1668d70b
Code: Unable to access opcode bytes at RIP 0x7f5a1668d6e1.
RSP: 002b:00007ffd5fc8fc78 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5a1668d70b
RDX: 000000000000797a RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006e89 R11: 0000000000000246 R12: 00007ffd5fc8fe98
R13: 0000000000000001 R14: 00007ffd5fc8fcd8 R15: 000000000000797a
INFO: task fallocate:58829 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:58829 ppid:9359 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fed462bd70b
Code: Unable to access opcode bytes at RIP 0x7fed462bd6e1.
RSP: 002b:00007ffcef0aa948 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fed462bd70b
RDX: 000000000000419f RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000003f17 R11: 0000000000000246 R12: 00007ffcef0aab68
R13: 0000000000000000 R14: 00007ffcef0aa9a8 R15: 000000000000419f
9[145778]: segfault at 8 ip 00007f6af4996875 sp 00007ffd28f350e0 error 4 in ld-2.28.so[7f6af4975000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[145928]: segfault at 8 ip 00007f2162293875 sp 00007ffcc986e220 error 4 in ld-2.28.so[7f2162272000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 0000000043db8782/0x3bd7133cabb44e35 lrc: 3/0,0 mode: PW/PW res: [0x300000402:0x196:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 20480->49151) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cabb44e27 expref: 4666 pid: 31650 timeout: 908 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6397:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792254916 with bad export cookie 4311936319525813210
Lustre: lustre-OST0000-osc-ffff980c0792e000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff980c0792e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0xfe6:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0xe72:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0xe6f:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x13bc:0x0]// may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x1004:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0xe0f:0x0]/ may get corrupted (rc -108)
LustreError: 154896:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff980c0792e000: namespace resource [0x300000402:0x1bf:0x0].0x0 (ffff980c052d0b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0000-osc-ffff980c0792e000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: ll_ost_io00_017: service thread pid 34319 was inactive for 524.126 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_017 state:D stack:0 pid:34319 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
15[157130]: segfault at 5573965c1000 ip 00005573965c1000 sp 00007fffd7770a38 error 14 in 15[5573967c1000+1000]
Code: Unable to access opcode bytes at RIP 0x5573965c0fd6.
systemd-coredump[157562]: Failed to get EXE, ignoring: No such process
11[160657]: segfault at 0 ip 000055cc7d74eb47 sp 00007fff894af0e0 error 6 in 11[55cc7d74a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: ll_ost_io00_005: service thread pid 10721 was inactive for 513.096 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_005 state:D stack:0 pid:10721 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: lustre-OST0001-osc-ffff980c05ae3800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-OST0001-osc-ffff980c05ae3800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x112a:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0xfae:0x0]/ may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x10c5:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x1165:0x0]/ may get corrupted (rc -108)
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 000000006dff3e28/0x3bd7133cabbf1f8e lrc: 3/0,0 mode: PW/PW res: [0x340000402:0x1c3:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 4096->32767) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cabbf1f87 expref: 19 pid: 15691 timeout: 976 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-ffff980c05ae3800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0002-osc-ffff980c05ae3800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x11ba:0x0]// may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x14ef:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0xfcc:0x0]/ may get corrupted (rc -108)
LustreError: 166484:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff980c05ae3800: namespace resource [0x340000402:0x1c3:0x0].0x0 (ffff980b87555b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 166484:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
Lustre: lustre-OST0002-osc-ffff980c05ae3800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0003-osc-ffff980c0792e000: granted 5758976 but already consumed 6840320
LustreError: lustre-OST0003-osc-ffff980c0792e000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x119a:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x1159:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x11a4:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x15d8:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x1161:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x15bc:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x15d2:0x0]// may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x11b6:0x0]/ may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x15fd:0x0]// may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x15ec:0x0]/ may get corrupted (rc -108)
Lustre: 21674:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff980c08735780 x1879314615781248/t0(0) o22->2dfe7373-8d79-477f-9fa5-b8053ca841db@0@lo:519/0 lens 440/432 e 12 to 0 dl 1792255004 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: lustre-OST0000-osc-ffff980c05ae3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x103c:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x16cc:0x0]// may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x11d2:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x1633:0x0]// may get corrupted (rc -108)
LustreError: 171195:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff980c05ae3800: namespace resource [0x300000402:0x211:0x0].0x0 (ffff980bcb255800) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 171195:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
16[170986]: segfault at 8 ip 00007f7ad839a875 sp 00007ffdfa757930 error 4 in ld-2.28.so[7f7ad8379000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 7745:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/4), not sending early reply req@ffff980bf49d7a80 x1879314621452544/t0(0) o22->2dfe7373-8d79-477f-9fa5-b8053ca841db@0@lo:563/0 lens 440/432 e 6 to 0 dl 1792255048 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 180931:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x280000404:0x1253:0x0] mdc close failed: rc = -13
0[181969]: segfault at 55715d89a000 ip 000055715d89a000 sp 00007ffc943643b8 error 14 in 0[55715da9a000+1000]
Code: Unable to access opcode bytes at RIP 0x55715d899fd6.
Lustre: ll_ost_io00_011: service thread pid 20441 was inactive for 554.864 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_011 state:R running task stack:0 pid:20441 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
6[183412]: segfault at 8 ip 00007fc12a37f875 sp 00007fff41700480 error 4 in ld-2.28.so[7fc12a35e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[186139]: segfault at 8 ip 00007f8b187a3875 sp 00007ffede4f2650 error 4 in ld-2.28.so[7f8b18782000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[186656]: segfault at 8 ip 00007f7f69244875 sp 00007ffc2122a190 error 4 in ld-2.28.so[7f7f69223000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[187546]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[187546]: Failed to get EXE, ignoring: No such process
Lustre: 3158:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792254448/real 1792254448] req@ffff980be2c81500 x1879314621452544/t0(0) o22->lustre-OST0001-osc-ffff980c0792e000@0@lo:6/4 lens 440/432 e 6 to 1 dl 1792255053 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0001: Client 2dfe7373-8d79-477f-9fa5-b8053ca841db (at 0@lo) reconnecting
Lustre: 9972:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff980c0a5ad780 x1879314628362240/t0(0) o22->2dfe7373-8d79-477f-9fa5-b8053ca841db@0@lo:620/0 lens 440/432 e 3 to 0 dl 1792255105 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 193321:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c0792e000: inode [0x240000404:0x1438:0x0] mdc close failed: rc = -13
Lustre: 3160:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792254505/real 1792254505] req@ffff980c0a5ac280 x1879314628362240/t0(0) o22->lustre-OST0002-osc-ffff980c0792e000@0@lo:6/4 lens 440/432 e 3 to 1 dl 1792255110 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0002-osc-ffff980c0792e000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0002: Client 2dfe7373-8d79-477f-9fa5-b8053ca841db (at 0@lo) reconnecting
Lustre: lustre-OST0002-osc-ffff980c0792e000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 3 previous similar messages
systemd-coredump[194575]: Failed to get EXE, ignoring: No such process
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000007758e869/0x3bd7133cabd5749d lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x1c3:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 12288->32767) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cabd57496 expref: 328 pid: 15691 timeout: 1126 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
LustreError: 15185:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff980c15ced080 x1879314731619328/t0(0) o104->lustre-OST0001@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 15185:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffff980c05ae3800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 4 previous similar messages
LustreError: lustre-OST0001-osc-ffff980c05ae3800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x1438:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x1206:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x14b8:0x0]// may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x132c:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x13f0:0x0]/ may get corrupted (rc -108)
LustreError: 196197:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff980c05ae3800: namespace resource [0x2c0000402:0x249:0x0].0x0 (ffff980bf7002500) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 196197:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 3 previous similar messages
Lustre: 18949:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 18949:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff980c05ae3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x1344:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x131b:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x153b:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x157f:0x0]// may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x1238:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x1a1d:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x1347:0x0]/ may get corrupted (rc -108)
LustreError: 15185:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff980bb47ddb00 x1879314736104960/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 11370:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792255165 with bad export cookie 4311936319531920577
LustreError: 11370:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) Skipped 1 previous similar message
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0002-osc-ffff980c05ae3800: granted 8437760 but already consumed 10416128
LustreError: lustre-OST0002-osc-ffff980c05ae3800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x1595:0x0]// may get corrupted (rc -5)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x1579:0x0]/ may get corrupted (rc -5)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x1483:0x0]// may get corrupted (rc -5)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x147e:0x0]/ may get corrupted (rc -5)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x1b0b:0x0]// may get corrupted (rc -5)
7[203139]: segfault at 8 ip 00007f969f3ad875 sp 00007ffe2da3dea0 error 4 in ld-2.28.so[7f969f38c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[203634]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[203634]: Failed to get EXE, ignoring: No such process
LustreError: 204380:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x200000403:0x1b0a:0x0] mdc close failed: rc = -13
19[205495]: segfault at 8 ip 00007fdd62d12875 sp 00007ffd582c1cb0 error 4 in ld-2.28.so[7fdd62cf1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[206311]: segfault at 8 ip 00007f7f870a2875 sp 00007ffe82643160 error 4 in ld-2.28.so[7f7f87081000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[206978]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[206978]: Failed to get EXE, ignoring: No such process
16[206601]: segfault at 8 ip 00007f4e6cf02875 sp 00007ffdffe550b0 error 4 in ld-2.28.so[7f4e6cee1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[207502]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[207502]: Failed to get EXE, ignoring: No such process
Lustre: 9972:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 9972:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 227 previous similar messages
Lustre: 9972:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9972:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 230 previous similar messages
Lustre: 9972:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 9972:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 230 previous similar messages
Lustre: 9972:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 9972:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 230 previous similar messages
Lustre: 9972:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9972:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 230 previous similar messages
Lustre: 9972:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 9972:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 230 previous similar messages
10[225856]: segfault at 0 ip 00005565a1fc7b47 sp 00007ffe54a9d030 error 6 in 10[5565a1fc3000+7000]
Code: Unable to access opcode bytes at RIP 0x5565a1fc7b1d.
systemd-coredump[226607]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[226607]: Failed to get EXE, ignoring: No such process
LustreError: 230444:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x200000404:0x1ddf:0x0] mdc close failed: rc = -13
12[243022]: segfault at 8 ip 00007f48cb9b2875 sp 00007fffefc2ef20 error 4 in ld-2.28.so[7f48cb991000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[276400]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[276400]: Failed to get EXE, ignoring: No such process
LustreError: 13140:0:(osc_cache.c:1007:osc_extent_wait()) extent ffff980b917852c0@{[3328 -> 3343/4095], [3|0|+|rpc|wihY|ffff980c0e858c40], [90112|16|+|-|ffff980c0459c800|1024|ffff980c0064cd00]} lustre-OST0001-osc-ffff980c0792e000: wait ext to 0 timedout, recovery in progress?
LustreError: 13140:0:(osc_cache.c:1007:osc_extent_wait()) ### extent: 0000000080c62cbe ns: lustre-OST0001-osc-ffff980c0792e000 lock: 000000009335c2b6/0x3bd7133cabdf0df8 lrc: 30/0,0 mode: PW/PW res: [0x2c0000402:0x249:0x0].0x0 rrc: 5 type: EXT [13631488->18446744073709551615] (req 13631488->13697023) gid 0 flags: 0x800429400020000 nid: local remote: 0x3bd7133cabdf0e68 expref: -99 pid: 187129 timeout: 0 lvb_type: 1 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 307866:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c0792e000: inode [0x200000403:0x2a62:0x0] mdc close failed: rc = -13
11[317600]: segfault at 8 ip 00007f9d2f314875 sp 00007ffd16c7a880 error 4 in ld-2.28.so[7f9d2f2f3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 151057:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 151057:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 229 previous similar messages
Lustre: 151057:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 151057:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 229 previous similar messages
Lustre: 151057:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 151057:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 229 previous similar messages
Lustre: 151057:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 151057:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 229 previous similar messages
Lustre: 151057:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 151057:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 229 previous similar messages
Lustre: 151057:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 151057:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 229 previous similar messages
0[329113]: segfault at 8 ip 00007fc28f07e875 sp 00007ffcaebbcbe0 error 4 in ld-2.28.so[7fc28f05d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3159:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792255124/real 1792255124] req@ffff980c08732a00 x1879314615781248/t0(0) o22->lustre-OST0002-osc-ffff980c0792e000@0@lo:6/4 lens 440/432 e 12 to 1 dl 1792255729 ref 1 fl Rpc:XQr/602/ffffffff rc -11/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0002-osc-ffff980c0792e000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0002: Client 2dfe7373-8d79-477f-9fa5-b8053ca841db (at 0@lo) reconnecting
Lustre: lustre-OST0002-osc-ffff980c0792e000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 3 previous similar messages
systemd-coredump[339204]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[339204]: Failed to get EXE, ignoring: No such process
LustreError: 341923:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x200000403:0x2fb3:0x0] mdc close failed: rc = -13
LustreError: 341923:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 6427:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3158:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792255095/real 1792255095] req@ffff980be2c81500 x1879314621452544/t0(0) o22->lustre-OST0001-osc-ffff980c0792e000@0@lo:6/4 lens 440/432 e 6 to 1 dl 1792255700 ref 1 fl Rpc:XQr/602/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0001: Client 2dfe7373-8d79-477f-9fa5-b8053ca841db (at 0@lo) reconnecting
LustreError: 350531:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_349214 LIST => FNAME no wakeup.
Lustre: 272451:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980c0e198700 x1879314745337856/t0(0) o22->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:752/0 lens 440/432 e 0 to 0 dl 1792255992 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 7745:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980bf4e13800 x1879314747895552/t0(0) o22->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:13/0 lens 440/432 e 0 to 0 dl 1792256008 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 350531:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_349214 LIST => FNAME no wakeup.
Lustre: ll_ost_io00_027: service thread pid 279620 was inactive for 42.829 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_027 state:R running task stack:0 pid:279620 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
ldiskfs_mark_inode_dirty+0x5e/0x390 [ldiskfs]
osd_fallocate_preallocate.isra.36+0x452/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
osd_fallocate+0x151/0x500 [osd_ldiskfs]
ofd_object_fallocate+0x83a/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 368190:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: 9972:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980c088cf380 x1879314755737856/t0(0) o22->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:70/0 lens 440/432 e 0 to 0 dl 1792256065 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 359865:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980c08741500 x1879314758428416/t0(0) o22->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:88/0 lens 440/432 e 0 to 0 dl 1792256083 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 20905:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980be06fd080 x1879314759544960/t0(0) o22->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:94/0 lens 440/432 e 0 to 0 dl 1792256089 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 384393:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: 3166:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792255334/real 1792255334] req@ffff980bcb1edb00 x1879314759544960/t0(0) o22->lustre-OST0001-osc-ffff980c05ae3800@0@lo:6/4 lens 440/432 e 0 to 1 dl 1792256094 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0001: Client 8e1db2fb-2dc6-4f97-8199-9e73d56e9154 (at 0@lo) reconnecting
Lustre: 64961:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980bcb29d080 x1879314768231296/t0(0) o22->2dfe7373-8d79-477f-9fa5-b8053ca841db@0@lo:154/0 lens 440/432 e 0 to 0 dl 1792256149 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 3164:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792255394/real 1792255394] req@ffff980be337e580 x1879314768231296/t0(0) o22->lustre-OST0003-osc-ffff980c0792e000@0@lo:6/4 lens 440/432 e 0 to 1 dl 1792256154 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0003: Client 2dfe7373-8d79-477f-9fa5-b8053ca841db (at 0@lo) reconnecting
LustreError: 405198:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c05ae3800: inode [0x200000404:0x3784:0x0] mdc close failed: rc = -13
LustreError: 405198:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: 3154:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792255095/real 1792255095] req@ffff980c08b4a300 x1879314725969792/t4294977171(4294977171) o4->lustre-OST0001-osc-ffff980c0792e000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1792255855 ref 2 fl Bulk:RXMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 3154:0:(client.c:2514:ptlrpc_expire_one_request()) Skipped 1 previous similar message
LustreError: 3154:0:(client.c:2385:ptlrpc_check_set()) @@@ bulk transfer failed 0/65536/0 req@ffff980c08b4a300 x1879314725969792/t4294977171(4294977171) o4->lustre-OST0001-osc-ffff980c0792e000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1792255855 ref 2 fl Bulk:ReXMQU/604/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 3154:0:(osc_request.c:2449:osc_brw_redo_request()) @@@ redo for recoverable error -5 req@ffff980c08b4a300 x1879314725969792/t4294977171(4294977171) o4->lustre-OST0001-osc-ffff980c0792e000@0@lo:6/4 lens 488/448 e 0 to 1 dl 1792255855 ref 2 fl Interpret:ReXMQU/604/0 rc -5/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 286s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000b466e0c1/0x3bd7133cac660668 lrc: 3/0,0 mode: PW/PW res: [0x380000402:0x491:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8192->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cac660653 expref: 6353 pid: 15185 timeout: 2071 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
LustreError: 196806:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff980c11709880 x1879314915840000/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 6398:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792256263 with bad export cookie 4311936319524928998
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0001-osc-ffff980c0792e000: granted 8437760 but already consumed 11051008
LustreError: lustre-OST0001-osc-ffff980c0792e000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: lustre-OST0003-osc-ffff980c0792e000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
LustreError: lustre-OST0003-osc-ffff980c0792e000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: lustre-OST0003-osc-ffff980c05ae3800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: lustre-OST0000-osc-ffff980c0792e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2bd6:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2e78:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2cf3:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3017:0x0]/ may get corrupted (rc -108)
LustreError: lustre-OST0002-osc-ffff980c05ae3800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2de5:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3056:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x306e:0x0]// may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2e28:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x38ff:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3a52:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3058:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2ee1:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2de5:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2dcc:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2d83:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2f15:0x0]// may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2dd3:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2dbf:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3b5e:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3b3a:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2d99:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3960:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2dd7:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2d34:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2e6e:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x30a1:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3b69:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2e90:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2f17:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2f52:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2d9e:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2dda:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3091:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2e5f:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x38e5:0x0]/ may get corrupted (rc -108)
LustreError: 416184:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff980c05ae3800: namespace resource [0x380000400:0x2b8:0x0].0x0 (ffff980bb4724500) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 416184:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x2fa6:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x383c:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2db8:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3032:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x2df1:0x0]// may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3855:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2f5b:0x0]/ may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x2f3f:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x2d79:0x0]/ may get corrupted (rc -108)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x30a2:0x0]/ may get corrupted (rc -108)
Lustre: ll_ost_io00_005: service thread pid 10721 completed after 1816.868s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3b88:0x0]// may get corrupted (rc -108)
Lustre: ll_ost_io00_027: service thread pid 279620 completed after 294.921s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
15[425067]: segfault at 8 ip 00007f82cc8f6875 sp 00007ffd85e18030 error 4 in ld-2.28.so[7f82cc8d5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[425722]: segfault at 8 ip 00007f2fa1321875 sp 00007fff1aadc570 error 4 in ld-2.28.so[7f2fa1300000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 359865:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/-150), not sending early reply req@ffff980bcb145080 x1879314792405504/t0(0) o22->8e1db2fb-2dc6-4f97-8199-9e73d56e9154@0@lo:315/0 lens 440/432 e 0 to 0 dl 1792256310 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
systemd-coredump[426673]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[426673]: Failed to get EXE, ignoring: No such process
Lustre: 3161:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792255555/real 1792255555] req@ffff980c21d8bf00 x1879314792405504/t0(0) o22->lustre-OST0000-osc-ffff980c05ae3800@0@lo:6/4 lens 440/432 e 0 to 1 dl 1792256315 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0000: Client 8e1db2fb-2dc6-4f97-8199-9e73d56e9154 (at 0@lo) reconnecting
Lustre: Skipped 1 previous similar message
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000034eb8e36/0x3bd7133cac980b44 lrc: 3/0,0 mode: PW/PW res: [0x380000402:0x557:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 24576->57343) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cac980b3d expref: 551 pid: 15696 timeout: 2367 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 4 previous similar messages
LustreError: lustre-OST0003-osc-ffff980c05ae3800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff980c05ae3800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 31580:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff980c082e1c00 x1879314934528256/t0(0) o104->lustre-OST0003@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: Skipped 10 previous similar messages
LustreError: 31580:0:(client.c:1404:ptlrpc_import_delay_req()) Skipped 4 previous similar messages
LustreError: lustre-OST0003-osc-ffff980c05ae3800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x30cd:0x0]/ may get corrupted (rc -5)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x3071:0x0]// may get corrupted (rc -5)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x30c2:0x0]/ may get corrupted (rc -5)
Lustre: 3157:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3de1:0x0]/ may get corrupted (rc -5)
Lustre: 3157:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x320f:0x0]/ may get corrupted (rc -5)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x30e1:0x0]/ may get corrupted (rc -5)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x30c1:0x0]/ may get corrupted (rc -5)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3b78:0x0]/ may get corrupted (rc -5)
LustreError: 440140:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff980c05ae3800: namespace resource [0x380000401:0x4f5:0x0].0x0 (ffff980bce3d1900) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 440140:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff980c05ae3800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 10 previous similar messages
LustreError: lustre-OST0000-osc-ffff980c05ae3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3182:0x0]/ may get corrupted (rc -5)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3293:0x0]// may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3e77:0x0]/ may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x30d8:0x0]/ may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3de3:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3e6a:0x0]// may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x3104:0x0]// may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x30a4:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x32b2:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x310a:0x0]/ may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3225:0x0]/ may get corrupted (rc -108)
LustreError: 442015:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff980c05ae3800: namespace resource [0x300000401:0x4d7:0x0].0x0 (ffff980bcc6e0700) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 442015:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0002-osc-ffff980c0792e000: granted 8437760 but already consumed 8986624
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) Skipped 1 previous similar message
LustreError: lustre-OST0002-osc-ffff980c0792e000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3cc9:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3406:0x0]// may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x33b2:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3d37:0x0]// may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3cf2:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x3e60:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3d3d:0x0]// may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3413:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x3052:0x0]// may get corrupted (rc -108)
Lustre: ll_ost_io00_017: service thread pid 34319 completed after 1989.178s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_011: service thread pid 20441 completed after 1888.298s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
systemd-coredump[449040]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[449040]: Failed to get EXE, ignoring: No such process
Lustre: 10514:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10514:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 335 previous similar messages
Lustre: 10514:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10514:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 336 previous similar messages
Lustre: 15175:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10514:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10514:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 336 previous similar messages
Lustre: 10514:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10514:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 336 previous similar messages
Lustre: 10514:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10514:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 336 previous similar messages
Lustre: 15175:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 338 previous similar messages
2[456122]: segfault at 8 ip 00007fc29d7f0875 sp 00007ffded40dfd0 error 4 in ld-2.28.so[7fc29d7cf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[456666]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[456666]: Failed to get EXE, ignoring: No such process
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000d0dd98fa/0x3bd7133cacaba22a lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x525:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 4096->16383) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133cacaba20e expref: 3505 pid: 11032 timeout: 2481 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
LustreError: 11370:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792256488 with bad export cookie 4311936319533395533
LustreError: 11370:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) Skipped 8 previous similar messages
LustreError: lustre-OST0001-osc-ffff980c05ae3800: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: lustre-OST0001-osc-ffff980c05ae3800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3615:0x0]/ may get corrupted (rc -5)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4188:0x0]// may get corrupted (rc -5)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x338a:0x0]/ may get corrupted (rc -5)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x34e5:0x0]// may get corrupted (rc -5)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4043:0x0]/ may get corrupted (rc -5)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x33c5:0x0]/ may get corrupted (rc -5)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3577:0x0]/ may get corrupted (rc -5)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x33cf:0x0]// may get corrupted (rc -5)
LustreError: lustre-OST0002-osc-ffff980c0792e000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x339a:0x0]/ may get corrupted (rc -108)
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x41e7:0x0]/ may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x33b0:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x3fbb:0x0]// may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x403e:0x0]// may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x368e:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x33ac:0x0]// may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x31e0:0x0]/ may get corrupted (rc -108)
LustreError: lustre-OST0003-osc-ffff980c0792e000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x4061:0x0]// may get corrupted (rc -5)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x404f:0x0]// may get corrupted (rc -5)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x36cf:0x0]// may get corrupted (rc -5)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x40b2:0x0]/ may get corrupted (rc -5)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x370d:0x0]// may get corrupted (rc -5)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x40a6:0x0]/ may get corrupted (rc -5)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4181:0x0]/ may get corrupted (rc -5)
Lustre: 3161:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x34e0:0x0]/ may get corrupted (rc -5)
LustreError: 466568:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff980c0792e000: namespace resource [0x380000402:0x5e0:0x0].0x0 (ffff980c13394400) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: lustre-OST0000-osc-ffff980c0792e000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffff980c0792e000: granted 8437760 but already consumed 9146368
LustreError: lustre-OST0000-osc-ffff980c0792e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x43b4:0x0]/ may get corrupted (rc -5)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x421f:0x0]/ may get corrupted (rc -5)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3769:0x0]/ may get corrupted (rc -5)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x34d7:0x0]/ may get corrupted (rc -5)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x43f2:0x0]/ may get corrupted (rc -5)
Lustre: 3155:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x43d2:0x0]/ may get corrupted (rc -5)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x38de:0x0]/ may get corrupted (rc -5)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x34ff:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x446e:0x0]// may get corrupted (rc -108)
LustreError: 481776:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff980c0792e000: namespace resource [0x300000402:0x605:0x0].0x0 (ffff980bb5e7ad00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 482323:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff980c0792e000: inode [0x200000404:0x42c0:0x0] mdc close failed: rc = -13
LustreError: 482323:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: lustre-OST0002-osc-ffff980c05ae3800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x35df:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x374f:0x0]/ may get corrupted (rc -108)
Lustre: 3160:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x36e3:0x0]/ may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x371d:0x0]// may get corrupted (rc -108)
Lustre: 3163:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x360e:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x423a:0x0]/ may get corrupted (rc -108)
19[488050]: segfault at 55852831a000 ip 000055852831a000 sp 00007ffea8b0a1b8 error 14 in 19[55852851a000+1000]
Code: Unable to access opcode bytes at RIP 0x558528319fd6.
16[495921]: segfault at 8 ip 00007fb05fbf0875 sp 00007ffefe0dc820 error 4 in ld-2.28.so[7fb05fbcf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000000db7d5e0/0x3bd7133caccbcf9b lrc: 3/0,0 mode: PW/PW res: [0x300000402:0x60c:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 28672->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3bd7133caccbcf78 expref: 517 pid: 62962 timeout: 2683 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6402:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 4 previous similar messages
LustreError: lustre-OST0000-osc-ffff980c0792e000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3ac1:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4704:0x0]/ may get corrupted (rc -108)
LustreError: 502490:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff980c0792e000: namespace resource [0x300000401:0x585:0x0].0x0 (ffff980c78281c00) refcount nonzero (2) after lock cleanup; forcing cleanup.
Lustre: 6427:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
LustreError: lustre-OST0002-osc-ffff980c05ae3800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4758:0x0]// may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3aea:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x382c:0x0]// may get corrupted (rc -108)
Lustre: 3159:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3b14:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3a47:0x0]/ may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x3850:0x0]/ may get corrupted (rc -108)
15[512330]: segfault at 8 ip 00007f486236d875 sp 00007ffc7b1b17f0 error 4 in ld-2.28.so[7f486234c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[512588]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[512588]: Failed to get EXE, ignoring: No such process
10[512914]: segfault at 55673286f000 ip 000055673286f000 sp 00007ffd86fda4b8 error 14 in 10[556732a6f000+1000]
Code: Unable to access opcode bytes at RIP 0x55673286efd6.
1[515506]: segfault at 0 ip 0000563b6fa5ba88 sp 00007fff65ff88e0 error 6 in 1[563b6fa5a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: lustre-OST0003-osc-ffff980c05ae3800: operation ost_punch to node 0@lo failed: rc = -107
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0003-osc-ffff980c05ae3800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3168:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x4975:0x0]/ may get corrupted (rc -5)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x49db:0x0]// may get corrupted (rc -108)
Lustre: 3164:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000403:0x49c9:0x0]// may get corrupted (rc -108)
Lustre: 3162:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3a9a:0x0]// may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x393f:0x0]/ may get corrupted (rc -108)
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000404:0x38dc:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3c54:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3b59:0x0]/ may get corrupted (rc -108)
Lustre: 3165:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3c5c:0x0]/ may get corrupted (rc -108)
Lustre: 18949:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 513 < left 520, rollback = 2
Lustre: 18949:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 3153:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffff980c05ae3800: granted 8437760 but already consumed 10579968
LustreError: lustre-OST0000-osc-ffff980c05ae3800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3166:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3b24:0x0]/ may get corrupted (rc -5)
Lustre: 3154:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000403:0x39aa:0x0]/ may get corrupted (rc -5)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x280000404:0x3c8c:0x0]/ may get corrupted (rc -108)
Lustre: 3169:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3b97:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x200000404:0x4827:0x0]/ may get corrupted (rc -108)
Lustre: 3167:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.87@tcp:/lustre/fid: [0x240000403:0x3b9b:0x0]/ may get corrupted (rc -108)
Lustre: 10891:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 515 < left 520, rollback = 2

LustreError: 272181:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 272181:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 8 PID: 272181 Comm: ll_sa_271680 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

11[7911]: segfault at 0 ip 00005615d533cb47 sp 00007ffe064b0fa0 error 6 in 11 (deleted)[5615d5338000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 9138:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9450411bd080 x1879232370981248/t4294970338(0) o101->06993aaf-ce3f-4387-9c49-eeb1c6c8705c@0@lo:488/0 lens 376/816 e 0 to 0 dl 1792175698 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6284:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6288:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6288:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6288:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6288:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6288:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6288:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6288:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6288:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6288:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6288:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6288:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6288:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6284:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 15575:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 6 previous similar messages
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 15575:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 7695:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7695:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 2 previous similar messages
Lustre: 7695:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7695:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 7695:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7695:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7695:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7695:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7695:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7695:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7695:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7695:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: ll_ost_io00_004: service thread pid 8610 was inactive for 41.869 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_004 state:R running task stack:0 pid:8610 ppid:2 flags:0x80004080
Call Trace:
? tgt_client_data_write.isra.13+0x1a6/0x210 [ptlrpc]
tgt_last_rcvd_update+0x39f/0xec0 [ptlrpc]
tgt_txn_stop_cb+0x123/0x680 [ptlrpc]
? jbd2_journal_extend+0x7d/0x2b0
? osd_declare_write+0x34a/0x7b0 [osd_ldiskfs]
? dt_declare_record_write+0x33/0x110 [obdclass]
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? osd_declare_attr_set+0x4ef/0x7c0 [osd_ldiskfs]
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? dt_declare_fallocate+0x30/0xa0 [obdclass]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 6288:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6288:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 6288:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6288:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6288:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6288:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6288:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 6288:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6288:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6288:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6288:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6288:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6284:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 6284:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6284:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6284:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6284:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6284:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6284:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 24417:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: ll_ost_io00_002: service thread pid 6289 was inactive for 74.944 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_002 state:R stack:0 pid:6289 ppid:2 flags:0x80004080
task:ll_ost_io00_006 state:R
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
running task stack:0 pid:14466 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
__getblk_gfp+0x59/0x5b0
? __es_insert_extent+0x2f5/0x550 [ldiskfs]
? ldiskfs_ext_map_blocks+0x2d6/0x1770 [ldiskfs]
__ldiskfs_get_inode_loc+0x10a/0x6f0 [ldiskfs]
? _raw_write_unlock+0x16/0x30
? osd_fallocate_preallocate.isra.36+0x452/0xc30 [osd_ldiskfs]
ldiskfs_reserve_inode_write+0x59/0x110 [ldiskfs]
? __might_sleep+0x60/0xc0
ldiskfs_mark_inode_dirty+0xa6/0x390 [ldiskfs]
osd_fallocate_preallocate.isra.36+0x452/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
osd_fallocate+0x151/0x500 [osd_ldiskfs]
ofd_object_fallocate+0x83a/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
hrtimer: interrupt took 7617543 ns
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 000000002b0a44bd/0xa8a0f482aa507634 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x11:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 28672->57343) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xa8a0f482aa50761f expref: 429 pid: 7892 timeout: 178 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 9512:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792175789 with bad export cookie 12150980636685812277
Lustre: lustre-OST0002-osc-ffff945063f63000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 5494:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff945007b25e80 x1879232392024960/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff945063f63000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xa0a:0x0]/ may get corrupted (rc -108)
Lustre: 3012:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x983:0x0]/ may get corrupted (rc -108)
Lustre: 3012:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xa7f:0x0]// may get corrupted (rc -108)
Lustre: 2998:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0xb00:0x0]/ may get corrupted (rc -108)
Lustre: 2999:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xa13:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff945063f63000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_004: service thread pid 8610 completed after 104.362s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000007b9a9277/0xa8a0f482aa533b64 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x27:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->24575) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xa8a0f482aa533b5d expref: 3329 pid: 7892 timeout: 190 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff945063f63000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff945063f63000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff945063f63000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3003:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xaa9:0x0]/ may get corrupted (rc -108)
Lustre: 3003:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xbbe:0x0]// may get corrupted (rc -108)
Lustre: 3011:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xb7a:0x0]/ may get corrupted (rc -108)
Lustre: 3004:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xb0c:0x0]/ may get corrupted (rc -108)
LustreError: 42532:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff945063f63000: namespace resource [0x280000400:0x2b:0x0].0x0 (ffff945007205e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff945063f63000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_002: service thread pid 6289 completed after 101.972s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000dd632423/0xa8a0f482aa53a218 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x2b:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 4096->28671) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0xa8a0f482aa53a203 expref: 558 pid: 6275 timeout: 192 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff94507a1df800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff94507a1df800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff94507a1df800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3009:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xb95:0x0]/ may get corrupted (rc -108)
Lustre: 3011:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0xb06:0x0]/ may get corrupted (rc -108)
Lustre: 3013:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xba0:0x0]/ may get corrupted (rc -108)
Lustre: 3013:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0xb10:0x0]/ may get corrupted (rc -108)
Lustre: 2998:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xbbe:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff94507a1df800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_006: service thread pid 14466 completed after 101.457s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 6289:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6289:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 6289:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6289:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 6289:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6289:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 6289:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6289:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 6289:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6289:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 6289:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6289:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 8213:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
9[47129]: segfault at 8 ip 00007f73f6d0e875 sp 00007ffc0ca02bb0 error 4 in ld-2.28.so[7f73f6ced000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[47669]: segfault at 8 ip 00007fdf6fc97875 sp 00007fff8b5bf090 error 4 in ld-2.28.so[7fdf6fc76000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[48120]: segfault at 8 ip 00007f8818f0d875 sp 00007ffeae0adf80 error 4 in ld-2.28.so[7f8818eec000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[57930]: segfault at 0 ip 0000558ad260eb47 sp 00007ffe1d78e7a0 error 6 in 17[558ad260a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 14466:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 14466:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 35 previous similar messages
Lustre: 14466:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14466:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 14466:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 14466:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 14466:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 14466:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 14466:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14466:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 14466:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14466:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 36 previous similar messages
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000c526b4f8/0xa8a0f482aa6bec56 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0xfd:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 28672->53247) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xa8a0f482aa6bec33 expref: 202 pid: 7892 timeout: 306 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff94507a1df800: operation ost_write to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff94507a1df800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff94507a1df800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3011:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x1966:0x0]// may get corrupted (rc -5)
Lustre: 3009:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x1643:0x0]// may get corrupted (rc -108)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x1799:0x0]/ may get corrupted (rc -108)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x18a6:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff94507a1df800: Connection restored to 0@lo (at 0@lo)
7[80258]: segfault at 8 ip 00007fc90d7b3875 sp 00007ffce0839230 error 4 in ld-2.28.so[7fc90d792000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000005990aa8/0xa8a0f482aa7039e1 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x12a:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 24576->28671) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0xa8a0f482aa7039be expref: 811 pid: 7892 timeout: 326 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff94507a1df800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff94507a1df800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff94507a1df800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 2998:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x17f8:0x0]/ may get corrupted (rc -5)
Lustre: 3012:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x1b85:0x0]// may get corrupted (rc -5)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x181e:0x0]/ may get corrupted (rc -5)
Lustre: 3004:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x1bb0:0x0]// may get corrupted (rc -5)
Lustre: 3011:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x19cf:0x0]/ may get corrupted (rc -5)
Lustre: 2998:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x19f0:0x0]/ may get corrupted (rc -5)
Lustre: 3012:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x19bb:0x0]/ may get corrupted (rc -5)
LustreError: 85310:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff94507a1df800: namespace resource [0x280000400:0x12a:0x0].0x0 (ffff94500c1e2e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 85310:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-ffff94507a1df800: Connection restored to 0@lo (at 0@lo)
18[90705]: segfault at 8 ip 00007efec06ab875 sp 00007ffc19b4e840 error 4 in ld-2.28.so[7efec068a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 000000000251df5f/0xa8a0f482aa772de7 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x160:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->36863) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xa8a0f482aa772c66 expref: 1269 pid: 52703 timeout: 354 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6386:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff945044f83480 x1879232430288000/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0003-osc-ffff945063f63000: operation ost_punch to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff945063f63000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff945063f63000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3004:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x1c80:0x0]/ may get corrupted (rc -108)
Lustre: 3009:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x1c2e:0x0]/ may get corrupted (rc -108)
Lustre: 3009:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x1ac6:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff945063f63000: Connection restored to 0@lo (at 0@lo)
5[99567]: segfault at 0 ip 0000557e9862fb47 sp 00007ffe5089a880 error 6 in 5[557e9862b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18[100135]: segfault at 8 ip 00007f15122ca875 sp 00007ffff98b1f10 error 4 in ld-2.28.so[7f15122a9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[100258]: segfault at 8 ip 00007f466e104875 sp 00007ffcf2e0c9c0 error 4 in ld-2.28.so[7f466e0e3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[102504]: segfault at 8 ip 00007fadc2b92875 sp 00007fff67ed4d20 error 4 in ld-2.28.so[7fadc2b71000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[105306]: segfault at 0 ip 0000555d354feb47 sp 00007ffc55b44760 error 6 in 8[555d354fa000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[105620]: segfault at 1 ip 0000557b2cc08950 sp 00007ffd68279848 error 6 in 8[557b2cc04000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[105587]: segfault at 563d08579f24 ip 0000563d08579f24 sp 00007fff20517a18 error 7 in 13[563d08575000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 7695:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7695:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 89 previous similar messages
Lustre: 7695:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7695:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 7695:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7695:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 7695:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7695:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 7695:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7695:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 7695:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7695:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 5522:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000002ea2707a/0xa8a0f482aa867749 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x1ce:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xa8a0f482aa8676e7 expref: 298 pid: 20739 timeout: 415 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffff94507a1df800: operation ost_fallocate to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff94507a1df800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff94507a1df800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
2[113705]: segfault at 8 ip 00007f0f3d38c875 sp 00007ffcb1dd7150 error 4 in ld-2.28.so[7f0f3d36b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3013:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x24fd:0x0]/ may get corrupted (rc -108)
Lustre: 2998:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x254b:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff94507a1df800: Connection restored to 0@lo (at 0@lo)
LustreError: 120490:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff94507a1df800: inode [0x200000401:0x26d8:0x0] mdc close failed: rc = -13
LustreError: 121227:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff945063f63000: inode [0x200000401:0x26d8:0x0] mdc close failed: rc = -13
LustreError: 121659:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff94507a1df800: inode [0x200000401:0x26d8:0x0] mdc close failed: rc = -13
LustreError: 126684:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff945063f63000: inode [0x200000402:0x289c:0x0] mdc close failed: rc = -13
15[130202]: segfault at 0 ip 00005590e29caa88 sp 00007fffb39e2e80 error 6 in 15[5590e29c9000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 24417:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000eff1c9dc/0xa8a0f482aaa3bfd9 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x2c5:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xa8a0f482aaa3bfa8 expref: 107 pid: 7892 timeout: 546 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff94507a1df800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff94507a1df800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff94507a1df800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3008:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x30c1:0x0]/ may get corrupted (rc -108)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x314a:0x0]// may get corrupted (rc -108)
Lustre: 2999:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x2fc2:0x0]// may get corrupted (rc -108)
Lustre: 3012:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x30f4:0x0]/ may get corrupted (rc -108)
Lustre: 3013:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x2fd8:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff94507a1df800: Connection restored to 0@lo (at 0@lo)
8[153082]: segfault at 8 ip 00007ff512069875 sp 00007fff217584a0 error 4 in ld-2.28.so[7ff512048000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 5522:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
LustreError: lustre-OST0001-osc-ffff945063f63000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-OST0001-osc-ffff945063f63000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3009:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x345d:0x0]// may get corrupted (rc -108)
Lustre: 3006:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x333a:0x0]/ may get corrupted (rc -108)
INFO: task fallocate:111965 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:111965 ppid:7482 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff47b68770b
Code: Unable to access opcode bytes at RIP 0x7ff47b6876e1.
RSP: 002b:00007fff04e3e338 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff47b68770b
RDX: 000000000000251d RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000049a5 R11: 0000000000000246 R12: 00007fff04e3e558
R13: 0000000000000000 R14: 00007fff04e3e398 R15: 000000000000251d
INFO: task chown:112074 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:112074 ppid:7421 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0cf3226fbe
Code: Unable to access opcode bytes at RIP 0x7f0cf3226f94.
RSP: 002b:00007ffc99afce08 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffc99afd040 RCX: 00007f0cf3226fbe
RDX: 00000000000035ad RSI: 0000559322316d20 RDI: 00000000ffffff9c
RBP: 0000559322317f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000004e07 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000559322317fc8 R14: 00000000000035ad R15: 0000000000004e07
INFO: task chown:112206 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:112206 ppid:7439 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f32ad4f1fbe
Code: Unable to access opcode bytes at RIP 0x7f32ad4f1f94.
RSP: 002b:00007ffe519b8348 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffe519b8580 RCX: 00007f32ad4f1fbe
RDX: 000000000000573b RSI: 0000563419686d20 RDI: 00000000ffffff9c
RBP: 0000563419687f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000000c1f R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000563419687fc8 R14: 000000000000573b R15: 0000000000000c1f
INFO: task fallocate:130949 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:130949 ppid:7500 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f6eafa9570b
Code: Unable to access opcode bytes at RIP 0x7f6eafa956e1.
RSP: 002b:00007ffd6e6c4488 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f6eafa9570b
RDX: 0000000000000dc1 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000041a9 R11: 0000000000000246 R12: 00007ffd6e6c46a8
R13: 0000000000000000 R14: 00007ffd6e6c44e8 R15: 0000000000000dc1
INFO: task chmod:131562 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:131562 ppid:7465 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5e4e0ea258
Code: Unable to access opcode bytes at RIP 0x7f5e4e0ea22e.
RSP: 002b:00007fff087df0b8 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5e4e0ea258
RDX: 0000000000000fd8 RSI: 00005640b28d7500 RDI: 00000000ffffff9c
RBP: 00005640b28d7470 R08: 0000000000000000 R09: 0000000000000fd8
R10: 0000000000000fff R11: 0000000000000246 R12: 00005640b24b0bb4
R13: 00005640b28d7500 R14: 00005640b28d8730 R15: 0000000000008000
Lustre: ll_ost_io00_005: service thread pid 11977 was inactive for 226.048 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
task:ll_ost_io00_005 state:R running task stack:0 pid:11977 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? kfree+0xef/0x380
? ldiskfs_map_blocks+0x723/0x920 [ldiskfs]
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? osd_fallocate_preallocate.isra.36+0x396/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x99a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 7695:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 7695:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 193 previous similar messages
Lustre: 7695:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7695:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 197 previous similar messages
Lustre: 7695:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7695:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 197 previous similar messages
Lustre: 7695:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 7695:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 197 previous similar messages
Lustre: 7695:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7695:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 197 previous similar messages
Lustre: 7695:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7695:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 197 previous similar messages
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 00000000bdefceb1/0xa8a0f482aab7cbff lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x354:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 20480->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xa8a0f482aab7cbf1 expref: 53 pid: 20722 timeout: 649 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff945063f63000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff945063f63000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff945063f63000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3006:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x3b6d:0x0]/ may get corrupted (rc -108)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x3b8c:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff945063f63000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff94507a1df800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3009:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x3e12:0x0]/ may get corrupted (rc -108)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x3c5c:0x0]/ may get corrupted (rc -108)
Lustre: 3013:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x3d5b:0x0]/ may get corrupted (rc -108)
Lustre: 3010:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x3c3a:0x0]/ may get corrupted (rc -108)
traps: 5[190035] general protection fault ip:565197dabf08 sp:7ffdb83c5710 error:0 in 5[565197da7000+7000]
Lustre: 5521:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 5521:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
19[195134]: segfault at 0 ip 000056165506bb47 sp 00007ffe7d9a5270 error 6 in 19[561655067000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
INFO: task fallocate:111965 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:111965 ppid:7482 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff47b68770b
Code: Unable to access opcode bytes at RIP 0x7ff47b6876e1.
RSP: 002b:00007fff04e3e338 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff47b68770b
RDX: 000000000000251d RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000049a5 R11: 0000000000000246 R12: 00007fff04e3e558
R13: 0000000000000000 R14: 00007fff04e3e398 R15: 000000000000251d
INFO: task chown:112074 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:112074 ppid:7421 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0cf3226fbe
Code: Unable to access opcode bytes at RIP 0x7f0cf3226f94.
RSP: 002b:00007ffc99afce08 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffc99afd040 RCX: 00007f0cf3226fbe
RDX: 00000000000035ad RSI: 0000559322316d20 RDI: 00000000ffffff9c
RBP: 0000559322317f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000004e07 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000559322317fc8 R14: 00000000000035ad R15: 0000000000004e07
INFO: task chown:112206 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:112206 ppid:7439 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f32ad4f1fbe
Code: Unable to access opcode bytes at RIP 0x7f32ad4f1f94.
RSP: 002b:00007ffe519b8348 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffe519b8580 RCX: 00007f32ad4f1fbe
RDX: 000000000000573b RSI: 0000563419686d20 RDI: 00000000ffffff9c
RBP: 0000563419687f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000000c1f R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000563419687fc8 R14: 000000000000573b R15: 0000000000000c1f
LustreError: 9512:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff94507072db00 x1879232510818816/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0001-osc-ffff945063f63000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3006:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x440a:0x0]/ may get corrupted (rc -108)
Lustre: 3004:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x43b5:0x0]/ may get corrupted (rc -108)
Lustre: 3004:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x436d:0x0]/ may get corrupted (rc -108)
18[207926]: segfault at 557223002000 ip 0000557223002000 sp 00007ffc05df55a8 error 14 in 18[557223202000+1000]
Code: Unable to access opcode bytes at RIP 0x557223001fd6.
6[212593]: segfault at 0 ip 000055e928facb47 sp 00007ffe454022c0 error 6 in 1[55e928fa8000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 5522:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000a2be4581/0xa8a0f482aade0622 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x4ab:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 28672->53247) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xa8a0f482aade061b expref: 713 pid: 20722 timeout: 832 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5495:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
LustreError: lustre-OST0003-osc-ffff945063f63000: operation ost_write to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff945063f63000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: 2997:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0003-osc-ffff945063f63000: granted 8437760 but already consumed 12840960
LustreError: lustre-OST0003-osc-ffff945063f63000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3002:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x4c70:0x0]// may get corrupted (rc -5)
Lustre: 3008:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x4b86:0x0]/ may get corrupted (rc -108)
Lustre: 3005:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x4c17:0x0]/ may get corrupted (rc -108)
Lustre: 3005:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x4c5f:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff945063f63000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 2 previous similar messages
INFO: task fallocate:111965 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:111965 ppid:7482 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff47b68770b
Code: Unable to access opcode bytes at RIP 0x7ff47b6876e1.
RSP: 002b:00007fff04e3e338 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff47b68770b
RDX: 000000000000251d RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000049a5 R11: 0000000000000246 R12: 00007fff04e3e558
R13: 0000000000000000 R14: 00007fff04e3e398 R15: 000000000000251d
INFO: task chown:112074 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:112074 ppid:7421 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0cf3226fbe
Code: Unable to access opcode bytes at RIP 0x7f0cf3226f94.
RSP: 002b:00007ffc99afce08 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffc99afd040 RCX: 00007f0cf3226fbe
RDX: 00000000000035ad RSI: 0000559322316d20 RDI: 00000000ffffff9c
RBP: 0000559322317f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000004e07 R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000559322317fc8 R14: 00000000000035ad R15: 0000000000004e07
8[240435]: segfault at 0 ip 000055cf2fe7bb47 sp 00007ffec9ad7af0 error 6 in 8[55cf2fe77000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[240776]: segfault at 8 ip 00007f9b94893875 sp 00007ffeb10a8e80 error 4 in ld-2.28.so[7f9b94872000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[248451]: segfault at 8 ip 00007fcc9665c875 sp 00007ffc8cb84030 error 4 in ld-2.28.so[7fcc9663b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[251407]: segfault at 8 ip 00007fc9ddccf875 sp 00007ffc0e5044b0 error 4 in ld-2.28.so[7fc9ddcae000+2f000]
18[251375]: segfault at 8 ip 00007f464f9af875 sp 00007ffed427c510 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7f464f98e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[254279]: segfault at 8 ip 00007f0df7f66875 sp 00007ffe09fa6870 error 4 in ld-2.28.so[7f0df7f45000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[254444]: segfault at 8 ip 00007f9882da7875 sp 00007ffe1688c920 error 4 in ld-2.28.so[7f9882d86000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48

LustreError: 37460:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 37460:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 10 PID: 37460 Comm: ll_sa_37207 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

15[8397]: segfault at 0 ip 00005643d1fdbb47 sp 00007ffc3415a8a0 error 6 in 15[5643d1fd7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4[8248]: segfault at 55e6f6a64000 ip 000055e6f6a64000 sp 00007ffc1edbe468 error 14 in 4 (deleted)[55e6f6c64000+1000]
Code: Unable to access opcode bytes at RIP 0x55e6f6a63fd6.
Lustre: 6266:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6266:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6266:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6266:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6266:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6266:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
traps: 13[11792] trap invalid opcode ip:558af12a1d6e sp:7ffc2c93a7b8 error:0 in 2[558af129b000+7000]
Lustre: 6268:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6268:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6268:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6268:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6268:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6268:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6268:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6268:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6268:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6268:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6268:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6268:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12967:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9a9fb5f43800 x1879188433034880/t4294971044(0) o101->a62b0ddb-0638-4fd0-8524-2711fe9fd397@0@lo:142/0 lens 376/816 e 0 to 0 dl 1792133827 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
8[14761]: segfault at 1b30 ip 0000000000001b30 sp 00007ffc426d40d0 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
Lustre: 15689:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 15689:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 15689:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15689:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15689:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 15689:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15689:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 15689:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15689:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15689:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 15689:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15689:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8826:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8826:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 8826:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8826:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8826:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8826:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8826:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8826:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8826:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8826:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8826:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8826:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6273:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6273:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 6273:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6273:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 6273:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6273:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 6273:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6273:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 6273:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6273:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 6273:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6273:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 9 previous similar messages
LustreError: 18467:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9a9fcd74d000: inode [0x200000401:0x178:0x0] mdc close failed: rc = -13
Lustre: 7979:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 7979:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 7979:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7979:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7979:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7979:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7979:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7979:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7979:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7979:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7979:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7979:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7979:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7979:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 7979:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7979:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 7979:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7979:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 7979:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7979:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 7979:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7979:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 7979:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7979:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 9 previous similar messages
9[28768]: segfault at 8 ip 00007f2b5b499875 sp 00007ffe70bbba40 error 4 in ld-2.28.so[7f2b5b478000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[28983]: segfault at 8 ip 00007f6746a55875 sp 00007ffc2ff19140 error 4 in ld-2.28.so[7f6746a34000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: ll_ost_io00_007: service thread pid 17081 was inactive for 42.663 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_007 state:R running task stack:0 pid:17081 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
ldiskfs_mark_inode_dirty+0x5e/0x390 [ldiskfs]
osd_fallocate_preallocate.isra.36+0x30d/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 8826:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8826:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 8826:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8826:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8826:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8826:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8826:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8826:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8826:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8826:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8826:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8826:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 5 previous similar messages

LustreError: 276372:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 276372:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 13 PID: 276372 Comm: ll_sa_276272 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 5527:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff90bdca7ef700 x1879145264101760/t4294967480(0) o101->dd26da97-add3-4698-ac2d-de006a807903@0@lo:471/0 lens 376/840 e 0 to 0 dl 1792092631 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8382:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8382:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8382:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8382:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8382:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8382:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8382:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 8382:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8382:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8382:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8382:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8382:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: ll_ost_io00_001: service thread pid 6294 was inactive for 43.909 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_001 state:R running task stack:0 pid:6294 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? ldiskfs_ext_map_blocks+0x2d6/0x1770 [ldiskfs]
? add_transaction_credits+0x20a/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
? jbd2__journal_start+0x11f/0x300
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? dt_trans_start+0x20/0x70 [obdclass]
? ofd_trans_start+0x7d/0xf0 [ofd]
? ofd_object_fallocate+0x6cf/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 8591:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8591:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 8591:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8591:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8591:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8591:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8591:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8591:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8591:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8591:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 8591:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8591:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6292:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6292:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 6292:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6292:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6292:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6292:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6292:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6292:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6292:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6292:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6292:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6292:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: ll_ost_io00_003: service thread pid 8144 was inactive for 74.931 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_003 state:R running task stack:0 pid:8144 ppid:2 flags:0x80004080
Call Trace:
? slab_free_freelist_hook.isra.49+0x111/0x230
? osd_trunc_unlock_all+0xf2/0x1e0 [osd_ldiskfs]
? kfree+0x28a/0x380
? _raw_read_unlock+0x16/0x30
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? kmem_cache_alloc_trace+0x5b/0x440
? osd_fallocate_preallocate.isra.36+0x396/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 000000003fd7cfa3/0x62d788fb6869879a lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x17:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 8192->24575) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x62d788fb686986cf expref: 19110 pid: 8065 timeout: 209 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 7911:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff90be43bb5e80 x1879145280061184/t0(0) o106->lustre-OST0003@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0003-osc-ffff90be46333000: operation ost_punch to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff90be46333000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff90be46333000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3027:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x735:0x0]/ may get corrupted (rc -108)
Lustre: 3033:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x6e7:0x0]/ may get corrupted (rc -108)
Lustre: 3033:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x71c:0x0]/ may get corrupted (rc -108)
Lustre: 3031:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x678:0x0]// may get corrupted (rc -108)
Lustre: 3029:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x61d:0x0]/ may get corrupted (rc -108)
LustreError: 29890:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff90be46333000: namespace resource [0x280000400:0x9e:0x0].0x0 (ffff90be254a2c00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: ll_ost_io00_003: service thread pid 8144 completed after 104.430s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-OST0003-osc-ffff90be46333000: Connection restored to 0@lo (at 0@lo)
Lustre: 12242:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12242:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 12242:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12242:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 12242:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12242:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 12242:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 12242:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 12242:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12242:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 12242:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12242:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 11 previous similar messages
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 0000000028a22964/0x62d788fb686c485c lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x31:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 12288->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62d788fb686c484e expref: 371 pid: 6271 timeout: 227 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff90bde292b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff90bde292b800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff90bde292b800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3019:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x750:0x0]/ may get corrupted (rc -108)
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x896:0x0]/ may get corrupted (rc -108)
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x8a3:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_001: service thread pid 6294 completed after 133.761s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 5541:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
15[36054]: segfault at 0 ip 00005566a2aa3b47 sp 00007ffe79fb1f30 error 6 in 15[5566a2a9f000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 00000000dc47c4cd/0x62d788fb686f5f71 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x4e:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->32767) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x62d788fb686f5e8a expref: 16274 pid: 6271 timeout: 253 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffff90bde292b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff90bde292b800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff90bde292b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x913:0x0]/ may get corrupted (rc -108)
LustreError: 38314:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff90bde292b800: namespace resource [0x240000400:0xd1:0x0].0x0 (ffff90be40c59800) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0000-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
Lustre: 6294:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6294:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 35 previous similar messages
Lustre: 6294:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6294:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 6294:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6294:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 6294:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6294:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 6294:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6294:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 6294:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6294:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 36 previous similar messages
0[44961]: segfault at 0 ip 000056147fd5ab47 sp 00007fffaacde100 error 6 in 0[56147fd56000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[50305]: segfault at 0 ip 00005573ca652b47 sp 00007ffee2b10c00 error 6 in 8[5573ca64e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000011051c36/0x62d788fb687c0116 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0xc0:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 24576->57343) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62d788fb687c00f3 expref: 13 pid: 6794 timeout: 333 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff90bde292b800: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff90bde292b800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff90bde292b800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0xedb:0x0]/ may get corrupted (rc -108)
Lustre: 3021:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0xf09:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
INFO: task fallocate:35767 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35767 ppid:7442 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbf6e92a70b
Code: Unable to access opcode bytes at RIP 0x7fbf6e92a6e1.
RSP: 002b:00007ffdd218d9f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fbf6e92a70b
RDX: 0000000000007a6b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000055f9 R11: 0000000000000246 R12: 00007ffdd218dc18
R13: 0000000000000000 R14: 00007ffdd218da58 R15: 0000000000007a6b
INFO: task chown:35953 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:35953 ppid:7460 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7212a51fbe
Code: Unable to access opcode bytes at RIP 0x7f7212a51f94.
RSP: 002b:00007ffdf4786b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffdf4786d40 RCX: 00007f7212a51fbe
RDX: 0000000000007266 RSI: 000055eba11e5d20 RDI: 00000000ffffff9c
RBP: 000055eba11e6f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000003de1 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000055eba11e6fc8 R14: 0000000000007266 R15: 0000000000003de1
INFO: task chmod:36004 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:36004 ppid:7503 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fe578e81258
Code: Unable to access opcode bytes at RIP 0x7fe578e8122e.
RSP: 002b:00007ffda26e3718 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe578e81258
RDX: 0000000000000e4a RSI: 000055a286650500 RDI: 00000000ffffff9c
RBP: 000055a286650470 R08: 0000000000000000 R09: 0000000000000e4a
R10: 0000000000000fff R11: 0000000000000246 R12: 000055a2847babb4
R13: 000055a286650500 R14: 000055a286651730 R15: 0000000000008000
11[68148]: segfault at 8 ip 00007f0fc5e87875 sp 00007ffc38b86060 error 4 in ld-2.28.so[7f0fc5e66000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6294:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6294:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 71 previous similar messages
Lustre: 6294:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6294:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 6294:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6294:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 6294:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6294:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 6294:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6294:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 6294:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6294:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 71 previous similar messages
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000077a2e279/0x62d788fb6889d6bb lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x12e:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62d788fb6889d6b4 expref: 22 pid: 6271 timeout: 437 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff90bde292b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff90bde292b800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff90bde292b800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3026:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x169b:0x0]/ may get corrupted (rc -108)
Lustre: 3024:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x1645:0x0]/ may get corrupted (rc -108)
Lustre: 3024:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x164e:0x0]/ may get corrupted (rc -108)
LustreError: 76839:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff90bde292b800: namespace resource [0x280000400:0x12e:0x0].0x0 (ffff90be2c26ff00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
Lustre: 5542:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
INFO: task fallocate:35767 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35767 ppid:7442 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 00000000108a37c8/0x62d788fb6892548f lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x18a:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->20479) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x62d788fb6892532a expref: 1904 pid: 8494 timeout: 489 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbf6e92a70b
Code: Unable to access opcode bytes at RIP 0x7fbf6e92a6e1.
RSP: 002b:00007ffdd218d9f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fbf6e92a70b
RDX: 0000000000007a6b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000055f9 R11: 0000000000000246 R12: 00007ffdd218dc18
R13: 0000000000000000 R14: 00007ffdd218da58 R15: 0000000000007a6b
INFO: task chown:35953 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:35953 ppid:7460 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7212a51fbe
Code: Unable to access opcode bytes at RIP 0x7f7212a51f94.
RSP: 002b:00007ffdf4786b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffdf4786d40 RCX: 00007f7212a51fbe
RDX: 0000000000007266 RSI: 000055eba11e5d20 RDI: 00000000ffffff9c
RBP: 000055eba11e6f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000003de1 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000055eba11e6fc8 R14: 0000000000007266 R15: 0000000000003de1
INFO: task chmod:36004 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:36004 ppid:7503 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fe578e81258
Code: Unable to access opcode bytes at RIP 0x7fe578e8122e.
RSP: 002b:00007ffda26e3718 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe578e81258
RDX: 0000000000000e4a RSI: 000055a286650500 RDI: 00000000ffffff9c
RBP: 000055a286650470 R08: 0000000000000000 R09: 0000000000000e4a
R10: 0000000000000fff R11: 0000000000000246 R12: 000055a2847babb4
R13: 000055a286650500 R14: 000055a286651730 R15: 0000000000008000
INFO: task fallocate:38301 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:38301 ppid:7473 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f576a0fa70b
Code: Unable to access opcode bytes at RIP 0x7f576a0fa6e1.
RSP: 002b:00007ffc1c57bc38 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f576a0fa70b
RDX: 0000000000006f01 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000007ebf R11: 0000000000000246 R12: 00007ffc1c57be58
R13: 0000000000000000 R14: 00007ffc1c57bc98 R15: 0000000000006f01
INFO: task fallocate:59612 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:59612 ppid:7489 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f863672470b
Code: Unable to access opcode bytes at RIP 0x7f86367246e1.
RSP: 002b:00007ffe719f0128 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f863672470b
RDX: 00000000000026a3 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005a95 R11: 0000000000000246 R12: 00007ffe719f0348
R13: 0000000000000000 R14: 00007ffe719f0188 R15: 00000000000026a3
Lustre: lustre-OST0000-osc-ffff90be46333000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff90be46333000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3021:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x19c4:0x0]// may get corrupted (rc -108)
Lustre: 3022:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x19a9:0x0]// may get corrupted (rc -108)
LustreError: 85283:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff90be46333000: namespace resource [0x240000400:0x1d6:0x0].0x0 (ffff90bdd758bf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 85283:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff90be46333000: Connection restored to 0@lo (at 0@lo)
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 0000000085c40c2f/0x62d788fb689a6e56 lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x1d6:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 16384->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62d788fb689a6e4f expref: 60 pid: 22823 timeout: 545 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffff90bde292b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff90bde292b800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff90bde292b800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3027:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x1c9c:0x0]/ may get corrupted (rc -108)
Lustre: 3024:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000401:0x1b7b:0x0]/ may get corrupted (rc -108)
LustreError: 94482:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff90bde292b800: namespace resource [0x240000400:0x1d5:0x0].0x0 (ffff90be32667e00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 94482:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
1[96944]: segfault at 8 ip 00007f69b45e5875 sp 00007ffde4896e30 error 4 in ld-2.28.so[7f69b45c4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: ll_ost_io00_003: service thread pid 8144 was inactive for 325.363 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_003 state:R running task stack:0 pid:8144 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? kmalloc_array+0x20/0x40 [ldiskfs]
preempt_schedule_common+0x2f/0x60
_cond_resched+0x3a/0x50
slab_pre_alloc_hook.constprop.64+0x138/0x1f0
? kmalloc_array+0x20/0x40 [ldiskfs]
__kmalloc+0x82/0x4c0
kmalloc_array+0x20/0x40 [ldiskfs]
ldiskfs_find_extent+0x392/0x450 [ldiskfs]
ldiskfs_ext_map_blocks+0x9b/0x1770 [ldiskfs]
? __might_sleep+0x60/0xc0
ldiskfs_map_blocks+0x2ac/0x920 [ldiskfs]
osd_fallocate_preallocate.isra.36+0x396/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
osd_fallocate+0x151/0x500 [osd_ldiskfs]
ofd_object_fallocate+0x83a/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_io00_007: service thread pid 12242 was inactive for 339.688 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_007 state:R running task stack:0 pid:12242 ppid:2 flags:0x80004080
Call Trace:
? __find_get_block+0x118/0x540
? ldiskfs_es_lookup_extent+0xd7/0x350 [ldiskfs]
? _cond_resched+0x21/0x50
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? get_page_from_freelist+0x1d9d/0x20f0
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? __ldiskfs_journal_get_write_access+0x64/0xd0 [ldiskfs]
? osd_ldiskfs_write+0x77b/0x830 [osd_ldiskfs]
? do_raw_spin_unlock+0x79/0x190
? __find_get_block+0x118/0x540
? check_eofblocks_fl+0x139/0x180 [ldiskfs]
? do_raw_spin_unlock+0x79/0x190
? dt_record_write+0x3f/0x190 [obdclass]
? allocate_slab+0x2c5/0x840
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? slab_free_freelist_hook.isra.49+0x111/0x230
? osd_declare_qid+0x40b/0x7c0 [osd_ldiskfs]
? do_raw_spin_unlock+0x79/0x190
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x36b/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: mdt00_007: service thread pid 7632 was inactive for 40.901 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: mdt00_020: service thread pid 28651 was inactive for 40.675 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
task:mdt00_007 state:I stack:0 pid:7632 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc27/0x1100 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x28c2/0x34c0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.48+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x34c0/0x34c0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
Lustre: Skipped 15 previous similar messages
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
INFO: task fallocate:35767 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35767 ppid:7442 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbf6e92a70b
Code: Unable to access opcode bytes at RIP 0x7fbf6e92a6e1.
RSP: 002b:00007ffdd218d9f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fbf6e92a70b
RDX: 0000000000007a6b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000055f9 R11: 0000000000000246 R12: 00007ffdd218dc18
R13: 0000000000000000 R14: 00007ffdd218da58 R15: 0000000000007a6b
INFO: task chown:35953 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:35953 ppid:7460 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7212a51fbe
Code: Unable to access opcode bytes at RIP 0x7f7212a51f94.
RSP: 002b:00007ffdf4786b08 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffdf4786d40 RCX: 00007f7212a51fbe
RDX: 0000000000007266 RSI: 000055eba11e5d20 RDI: 00000000ffffff9c
RBP: 000055eba11e6f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000003de1 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000055eba11e6fc8 R14: 0000000000007266 R15: 0000000000003de1
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: 00000000ca84ce64/0x62d788fb68abe4a8 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e0b:0x0].0x0 bits 0x19/0x0 rrc: 5 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x62d788fb68abe41c expref: 940 pid: 7671 timeout: 665 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: mdt_io00_004: service thread pid 79802 completed after 102.724s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 7593 completed after 102.174s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 7632:0:(ldlm_lockd.c:1445:ldlm_handle_enqueue()) ### lock on destroyed export 000000005229e977 ns: mdt-lustre-MDT0000_UUID lock: 0000000072212a7b/0x62d788fb68abe7b1 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 21 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x62d788fb68abe79c expref: 20 pid: 7632 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
Lustre: mdt00_020: service thread pid 28651 completed after 102.204s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 7628 completed after 102.677s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 40129 completed after 102.431s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff90bde292b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff90bde292b800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_011: service thread pid 8225 completed after 102.230s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 5529 completed after 102.444s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 9699 completed after 102.489s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_013: service thread pid 9025 completed after 102.425s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_015: service thread pid 9726 completed after 102.227s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 8220 completed after 102.570s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 7632 completed after 102.455s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 8222 completed after 102.437s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_008: service thread pid 7634 completed after 102.242s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_019: service thread pid 24737 completed after 102.439s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 7629 completed after 102.248s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff90bde292b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_001: service thread pid 5528 completed after 102.243s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 98673:0:(file.c:6307:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
Lustre: mdt00_012: service thread pid 8235 completed after 102.332s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 98488:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff90bde292b800: inode [0x200000401:0x1e17:0x0] mdc close failed: rc = -108
Lustre: mdt_out00_002: service thread pid 7671 completed after 103.022s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 98488:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 98879:0:(file.c:6307:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 98879:0:(file.c:6307:ll_inode_revalidate_fini()) Skipped 63 previous similar messages
LustreError: 98796:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff90bde292b800: inode [0x200000401:0x1dd3:0x0] mdc close failed: rc = -108
LustreError: 98796:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 585, rollback = 7
Lustre: 6296:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6296:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6296:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/585/0, punch: 0/0/0, quota 4/150/0
Lustre: 6296:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6296:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6296:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: ll_ost_io00_009: service thread pid 24843 was inactive for 335.471 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 2 previous similar messages
Lustre: ll_ost00_010: service thread pid 22814 was inactive for 40.102 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: 9639:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
0[110642]: segfault at 0 ip 0000560b823e8b47 sp 00007ffffc1b3f60 error 6 in 0[560b823e4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 5542:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000007d7cb24f/0x62d788fb688d7e53 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x152:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8192->36863) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x62d788fb688d7e0d expref: 9049 pid: 6272 timeout: 769 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5513:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792093303 with bad export cookie 7122311949079471004
LustreError: lustre-OST0001-osc-ffff90bde292b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff90bde292b800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: ll_ost00_010: service thread pid 22814 completed after 101.675s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: Skipped 5 previous similar messages
Lustre: ll_ost_io00_009: service thread pid 24843 completed after 413.444s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-OST0001-osc-ffff90bde292b800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3033:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x6ca:0x0]// may get corrupted (rc -108)
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x58a:0x0]/ may get corrupted (rc -108)
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x6cd:0x0]/ may get corrupted (rc -108)
Lustre: 3023:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x6dc:0x0]/ may get corrupted (rc -108)
Lustre: 3018:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x451:0x0]/ may get corrupted (rc -108)
Lustre: 3019:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x4d5:0x0]/ may get corrupted (rc -108)
Lustre: 3023:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x23db:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
Lustre: 5540:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
LustreError: lustre-OST0002-osc-ffff90bde292b800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3025:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x7b7:0x0]/ may get corrupted (rc -108)
LustreError: 120465:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff90bde292b800: namespace resource [0x2c0000400:0x253:0x0].0x0 (ffff90bdd7d30d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
1[126003]: segfault at 0 ip 000055b78ed7fb47 sp 00007ffda870b130 error 6 in 1[55b78ed7b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 77025:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff90be5152aa00 x1879145284180352/t0(0) o22->dd26da97-add3-4698-ac2d-de006a807903@0@lo:453/0 lens 440/432 e 16 to 0 dl 1792093368 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
19[132327]: segfault at 1 ip 000055fb9bdd3950 sp 00007ffe42ef5958 error 6 in 19[55fb9bdcf000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 77025:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/4), not sending early reply req@ffff90be36a7d400 x1879145285820416/t0(0) o22->dd26da97-add3-4698-ac2d-de006a807903@0@lo:472/0 lens 440/432 e 16 to 0 dl 1792093387 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: lustre-OST0001-osc-ffff90bde292b800: operation ost_setattr to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: lustre-OST0001-osc-ffff90bde292b800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3020:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0xda3:0x0]/ may get corrupted (rc -108)
LustreError: 137734:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff90bde292b800: namespace resource [0x300000400:0x33a:0x0].0x0 (ffff90be54ddbf00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 137734:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 79802:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 520, rollback = 2
Lustre: 79802:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 000000007de5c1cc/0x62d788fb68c68328 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x330:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->32767) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x62d788fb68c682e2 expref: 1056 pid: 6273 timeout: 960 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5517:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff90bde292b800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: lustre-OST0003-osc-ffff90bde292b800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3033:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x2ece:0x0]/ may get corrupted (rc -108)
Lustre: 3033:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x1251:0x0]/ may get corrupted (rc -108)
Lustre: 3018:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000403:0x1195:0x0]/ may get corrupted (rc -108)
Lustre: 3019:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.22@tcp:/lustre/fid: [0x200000402:0x2f9a:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff90bde292b800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 2 previous similar messages
6[154622]: segfault at 55ca9404ff24 ip 000055ca9404ff24 sp 00007ffc3cb25f28 error 7 in 6[55ca9404b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[157048]: segfault at 8 ip 00007f324542e875 sp 00007ffc5d2bf040 error 4 in ld-2.28.so[7f324540d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[157822]: segfault at 8 ip 00007fc27cbbb875 sp 00007ffeb31230a0 error 4 in ld-2.28.so[7fc27cb9a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[158260]: segfault at 8 ip 00007f7b23890875 sp 00007ffd5fdc2db0 error 4 in ld-2.28.so[7f7b2386f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3024:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792092768/real 1792092768] req@ffff90bdd037f380 x1879145284180352/t0(0) o22->lustre-OST0002-osc-ffff90be46333000@0@lo:6/4 lens 440/432 e 16 to 1 dl 1792093373 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0002: Client dd26da97-add3-4698-ac2d-de006a807903 (at 0@lo) reconnecting
Lustre: 3023:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792092787/real 1792092787] req@ffff90be3a4dfa80 x1879145285820416/t0(0) o22->lustre-OST0003-osc-ffff90be46333000@0@lo:6/4 lens 440/432 e 16 to 1 dl 1792093392 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0003: Client dd26da97-add3-4698-ac2d-de006a807903 (at 0@lo) reconnecting
ptlrpc_watchdog_fire: 20 callbacks suppressed
Lustre: ll_ost_io00_011: service thread pid 66379 was inactive for 581.491 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_011 state:R running task stack:0 pid:66379 ppid:2 flags:0x80004080
Call Trace:
? kfree+0x28a/0x380
? ldiskfs_ext_map_blocks+0x2d6/0x1770 [ldiskfs]
? __es_find_extent_range+0x128/0x1f0 [ldiskfs]
? tgt_last_rcvd_update+0x39f/0xec0 [ptlrpc]
? start_this_handle+0x720/0x850
? __might_sleep+0x60/0xc0
? ldiskfs_map_blocks+0x2ac/0x920 [ldiskfs]
? osd_fallocate_preallocate.isra.36+0x396/0xc30 [osd_ldiskfs]
? osd_attr_get+0x190/0x8b0 [osd_ldiskfs]
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 11671:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (2/2), not sending early reply req@ffff90be3a70bf00 x1879145320101504/t0(0) o22->dd26da97-add3-4698-ac2d-de006a807903@0@lo:721/0 lens 440/432 e 1 to 0 dl 1792093636 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 3032:0:(client.c:2514:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792093036/real 1792093036] req@ffff90bdd1a1c280 x1879145320101504/t0(0) o22->lustre-OST0000-osc-ffff90be46333000@0@lo:6/4 lens 440/432 e 1 to 1 dl 1792093641 ref 1 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0000: Client dd26da97-add3-4698-ac2d-de006a807903 (at 0@lo) reconnecting
Lustre: 24843:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 579, rollback = 7
Lustre: 24843:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 195 previous similar messages
Lustre: 24843:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 24843:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 200 previous similar messages
Lustre: 24843:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 24843:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 200 previous similar messages
Lustre: 24843:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/579/0, punch: 0/0/0, quota 1/3/0
Lustre: 24843:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 200 previous similar messages
Lustre: 24843:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 24843:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 200 previous similar messages
Lustre: 24843:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 24843:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 200 previous similar messages
LustreError: 197703:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff90be46333000: inode [0x200000403:0x21a3:0x0] mdc close failed: rc = -13
LustreError: 197703:0:(file.c:252:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 198375:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff90be46333000: inode [0x200000403:0x21d7:0x0] mdc close failed: rc = -13
7[206310]: segfault at 8 ip 00007fbdbede5875 sp 00007ffd7e04eba0 error 4 in ld-2.28.so[7fbdbedc4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[209272]: segfault at 8 ip 00007fcf26f8f875 sp 00007ffd2a9998b0 error 4 in ld-2.28.so[7fcf26f6e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 5540:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
LustreError: 238681:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff90be46333000: inode [0x200000403:0x2e3e:0x0] mdc close failed: rc = -13
Lustre: 'ost_io' is processing requests too slowly, client may timeout. Late by 15s, missed 1 early replies (reqs waiting=0 active=6, at_estimate=600, delay=1ms)
Lustre: 77025:0:(service.c:1570:ptlrpc_at_send_early_reply()) @@@ Already past deadline (-15s), not sending early reply. Consider increasing at_early_margin (5)? req@ffff90be34a0fa80 x1879145345566592/t0(0) o22->edcbf1ba-a4b1-453d-b2a4-6eab9eacf11e@0@lo:392/0 lens 440/432 e 0 to 0 dl 1792094062 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
18[249725]: segfault at 8 ip 00007f2a74530875 sp 00007ffd8a40ae20 error 4 in ld-2.28.so[7f2a7450f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[249840]: segfault at 8 ip 00007f9f66e6e875 sp 00007ffcd57a0b80 error 4 in ld-2.28.so[7f9f66e4d000+2f000]
18[249872]: segfault at 8 ip 00007f3bc6794875 sp 00007ffd44dd9fe0 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7f3bc6773000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48

LustreError: 294824:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 294824:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 11 PID: 294824 Comm: ll_sa_291940 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? _raw_spin_unlock_irqrestore+0x2f/0x60
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

8[13787]: segfault at 8 ip 00007fc22b20e875 sp 00007fffb598b330 error 4 in ld-2.28.so[7fc22b1ed000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11773:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fe01634f000 x1878932433560832/t4294968074(0) o101->40f3b05e-68fb-4c20-a769-a7b8ea3de5f4@0@lo:616/0 lens 376/816 e 0 to 0 dl 1791889681 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 15340:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fe01eb81c00 x1878932434041984/t4294968060(0) o101->40f3b05e-68fb-4c20-a769-a7b8ea3de5f4@0@lo:620/0 lens 376/840 e 0 to 0 dl 1791889685 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
16[17211]: segfault at 8 ip 00007f0d5c552875 sp 00007ffeff868180 error 4 in ld-2.28.so[7f0d5c531000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[18303]: segfault at 8 ip 00007f25e1518875 sp 00007fffd4659020 error 4 in ld-2.28.so[7f25e14f7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[18002]: segfault at 8 ip 00007f58a4338875 sp 00007ffece6fcd90 error 4 in ld-2.28.so[7f58a4317000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[30109]: segfault at 0 ip 0000563502590b47 sp 00007ffc635eac50 error 6 in 7[56350258c000+7000]
Code: 1f 00 e8 8c cb ff ff 85 c0 74 d0 48 85 ed 74 1a 48 01 5d 00 eb 14 0f 1f 44 00 00 48 89 ee 4c 89 e7 48 89 1c 24 e8 69 cb ff 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14[31824]: segfault at 8 ip 00007f1bdc0e0875 sp 00007fffbdfec220 error 4 in ld-2.28.so[7f1bdc0bf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[33680]: segfault at 557f4d8b9000 ip 0000557f4d8b9000 sp 00007ffdb0a53bc8 error 14 in 15[557f4dab9000+1000]
Code: Unable to access opcode bytes at RIP 0x557f4d8b8fd6.
traps: 5[38479] trap invalid opcode ip:560a6c84edac sp:7ffeb4c1c368 error:0 in 5[560a6c849000+7000]
13[38356]: segfault at 8 ip 00007f9c2e301875 sp 00007ffc4bf7ad00 error 4 in ld-2.28.so[7f9c2e2e0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 34217:0:(statahead.c:2416:start_statahead_thread()) lustre: invalid pattern 0X0.
Lustre: 15737:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9fe048a73f00 x1878932452432512/t4294977557(0) o101->40f3b05e-68fb-4c20-a769-a7b8ea3de5f4@0@lo:736/0 lens 376/840 e 0 to 0 dl 1791889801 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
6[55637]: segfault at 8 ip 00007f709ba8e875 sp 00007ffea7bdf1c0 error 4 in ld-2.28.so[7f709ba6d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 15771:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x625:0x0] ACL: rc = -2
5[59173]: segfault at 74 ip 000055beecc9c08d sp 00007ffc27abd8c0 error 6 in 5[55beecc97000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 20 4e 55 4d 42 45 52 5b 53 55 46 46 49 58 5d 2e 2e 2e 0a 20 <20> 6f 72 3a 20 20 25 73 20 4f 50 54 49 4f 4e 0a 50 61 75 73 65 20
18[60973]: segfault at 0 ip 0000564260f4035c sp 00007ffe4f9a1d88 error 6 in 18[564260f3b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 f5 40 38 2e 33 30 <00> 43 00 77 72 69 74 65 20 65 72 72 6f 72 00 25 73 3a 20 25 73 00
17[61564]: segfault at 8 ip 00007f12ea07e875 sp 00007fffdb899130 error 4 in ld-2.28.so[7f12ea05d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 12340:0:(out_handler.c:886:out_tx_end()) lustre-MDT0000-osd: error during execution of #0 from /home/green/git/lustre-release/lustre/ptlrpc/../target/out_handler.c:535: rc = -116
LustreError: 12101:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x6da:0x0] ACL: rc = -2
3[69123]: segfault at 8 ip 00007f8661970875 sp 00007ffddc4103b0 error 4 in ld-2.28.so[7f866194f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[81544]: segfault at 8 ip 00007fecaeec5875 sp 00007fff11a30520 error 4 in ld-2.28.so[7fecaeea4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[82982]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[82982]: Failed to get EXE, ignoring: No such process
LustreError: 86991:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fdfc5ca3000: inode [0x240000404:0x6f1:0x0] mdc close failed: rc = -13
traps: 10[92900] trap invalid opcode ip:5568fd632e64 sp:7ffea1ba2ec0 error:0 in 10[5568fd630000+7000]
traps: 10[93746] trap invalid opcode ip:558365af6e64 sp:7fffe1f8ae10 error:0 in 10[558365af4000+7000]
1[100946]: segfault at 8 ip 00007ff7c4c21875 sp 00007ffc5cd8c330 error 4 in ld-2.28.so[7ff7c4c00000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[106227]: segfault at 0 ip 00005644851e2b47 sp 00007ffe4fcd4830 error 6 in 19[5644851de000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 19[111394] general protection fault ip:55675aa97bba sp:7fff2da31688 error:0 in 19[55675aa92000+7000]
ODEBUG: object 0000000054714d4d is on stack 0000000041b5a805, but NOT annotated.
WARNING: CPU: 7 PID: 11525 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 7 PID: 11525 Comm: mdt00_009 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: de 88 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffab8be0f17338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffab8be0f17470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9fe1b23de5a8 RDI: ffff9fe1b23de5a8
RBP: ffffab8be0f17388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffab8be0f17108 R12: ffffffff8a8914e8
R13: ffffffff89505aa0 R14: 000000000000c120 R15: ffff9fdf8ea194b0
FS: 0000000000000000(0000) GS:ffff9fe1b23c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fc9515ff018 CR3: 00000001f4786000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1164/0x2560 [lod]
? slab_post_alloc_hook+0x6d/0x390
? __check_object_size+0x10e/0x26d
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
lod_declare_create+0x237/0xc30 [lod]
dt_declare_create+0x79/0x230 [obdclass]
? lod_alloc_comp_entries+0x28e/0x640 [lod]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x6d7/0x2550 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 7b7f189b6aa3308f ]---
19[118911]: segfault at 0 ip 00005565efd6fa88 sp 00007ffe57bec350 error 6 in 19[5565efd6e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[118409]: segfault at 8 ip 00007ffa35f36875 sp 00007ffe23586ab0 error 4 in ld-2.28.so[7ffa35f15000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[123065]: segfault at 0 ip 000056327c98fa88 sp 00007fffa56d2520 error 6 in 12[56327c98e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[123941]: segfault at 7ffd26fc5ca2 ip 00007ffd26fc5ca2 sp 00007ffd26fc34e0 error 15
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <2f> 6d 6e 74 2f 6c 75 73 74 72 65 2f 72 61 63 65 72 32 2f 31 33 00
LustreError: 125432:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fdfeac4d800: inode [0x200000403:0x106e:0x0] mdc close failed: rc = -13
1[126270]: segfault at 0 ip 0000557c3347fb47 sp 00007ffd7800ee50 error 6 in 1[557c3347b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 127255:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fdfc5ca3000: inode [0x200000403:0x106e:0x0] mdc close failed: rc = -13
LustreError: 51977:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x113d:0x0] ACL: rc = -2
1[132764]: segfault at 8 ip 00007f049fba0875 sp 00007fff1908bd70 error 4 in ld-2.28.so[7f049fb7f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[136537]: segfault at 8 ip 00007fc3b9e70875 sp 00007ffec7e0f620 error 4 in ld-2.28.so[7fc3b9e4f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 36390:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0xeb5:0x0] ACL: rc = -2
9[147199]: segfault at 8 ip 00007fab76c07875 sp 00007ffe1328d1e0 error 4 in ld-2.28.so[7fab76be6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 9025:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x1603:0x0] ACL: rc = -2
0[155524]: segfault at 1 ip 000055bea9503950 sp 00007ffdf66e8668 error 6 in 0[55bea94ff000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 89 d5 41 54 55 48 89 f5 53 48
2[164601]: segfault at 0 ip 000055c018505200 sp 00007ffda1eac118 error 6 in 2[55c018503000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14[168428]: segfault at 1b30 ip 0000000000001b30 sp 00007fff8d8c0b60 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
18[170224]: segfault at 0 ip 0000556a26b88b47 sp 00007fff5ebc0e10 error 6 in 18[556a26b84000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 172235:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fdfeac4d800: inode [0x240000404:0x10a8:0x0] mdc close failed: rc = -13
LustreError: 15250:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x122a:0x0] ACL: rc = -2
7[177154]: segfault at 55de626c3000 ip 000055de626c3000 sp 00007fff27cfb9a0 error 14 in 7[55de628c3000+1000]
Code: Unable to access opcode bytes at RIP 0x55de626c2fd6.
9[187142]: segfault at 8 ip 00007f56edf4b875 sp 00007ffe4b2a0b10 error 4 in ld-2.28.so[7f56edf2a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[188088]: segfault at 8 ip 00007f4c37849875 sp 00007fff3d07cce0 error 4 in ld-2.28.so[7f4c37828000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[194010]: segfault at 8 ip 00007ffb9827a875 sp 00007ffefdb3dd10 error 4 in ld-2.28.so[7ffb98259000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ODEBUG: object 0000000080338fd6 is on stack 00000000c5d40761, but NOT annotated.
WARNING: CPU: 13 PID: 12116 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 13 PID: 12116 Comm: mdt00_021 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: de 88 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffab8be25c7338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffab8be25c7470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9fe1b255e5a8 RDI: ffff9fe1b255e5a8
RBP: ffffab8be25c7388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffab8be25c7108 R12: ffffffff8a8cc028
R13: ffffffff89505aa0 R14: 0000000000046c60 R15: ffff9fdfffef0690
FS: 0000000000000000(0000) GS:ffff9fe1b2540000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fffe0badff8 CR3: 00000001054ab000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1164/0x2560 [lod]
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
lod_declare_create+0x237/0xc30 [lod]
dt_declare_create+0x79/0x230 [obdclass]
? lod_alloc_comp_entries+0x28e/0x640 [lod]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x6d7/0x2550 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 7b7f189b6aa33090 ]---
4[203968]: segfault at 55fac93bf000 ip 000055fac93bf000 sp 00007fffb1ce8c28 error 14 in 4[55fac95bf000+1000]
Code: Unable to access opcode bytes at RIP 0x55fac93befd6.
4[204109]: segfault at 55836a4c7000 ip 000055836a4c7000 sp 00007fffbcc859f8 error 14 in 4[55836a6c7000+1000]
Code: Unable to access opcode bytes at RIP 0x55836a4c6fd6.
0[208172]: segfault at 8 ip 00007f23c9529875 sp 00007ffe3504a4d0 error 4 in ld-2.28.so[7f23c9508000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[210264]: segfault at 8 ip 00007fa2bd8cf875 sp 00007ffe513f6210 error 4 in ld-2.28.so[7fa2bd8ae000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[213555]: segfault at 8 ip 00007fcd3d162875 sp 00007ffc655add80 error 4 in ld-2.28.so[7fcd3d141000+2f000]
5[212960]: segfault at 8 ip 00007fc78e5fd875 sp 00007ffd5d820830 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7fc78e5dc000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 36390:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x1f68:0x0] ACL: rc = -2
ODEBUG: object 00000000a4e3a7bd is on stack 00000000e83aa3f5, but NOT annotated.
WARNING: CPU: 4 PID: 11740 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 4 PID: 11740 Comm: mdt00_011 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: de 88 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffab8be150f338 EFLAGS: 00010006
RAX: 0000000000000050 RBX: ffffab8be150f470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9fe1b231e5a8 RDI: ffff9fe1b231e5a8
RBP: ffffab8be150f388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffab8be150f108 R12: ffffffff8a8e3c08
R13: ffffffff89505aa0 R14: 000000000005e840 R15: ffff9fe0035e3190
FS: 0000000000000000(0000) GS:ffff9fe1b2300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005584d2069c28 CR3: 00000001ff3cc000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1164/0x2560 [lod]
? _cond_resched+0x21/0x50
? __mutex_lock.isra.10+0x315/0xec0
? __check_object_size+0x10e/0x26d
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? __osd_sa_xattr_get.isra.8+0xb3/0x290 [osd_zfs]
lod_declare_create+0x237/0xc30 [lod]
dt_declare_create+0x79/0x230 [obdclass]
? lod_alloc_comp_entries+0x28e/0x640 [lod]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x6d7/0x2550 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 7b7f189b6aa33091 ]---
19[222361]: segfault at 8 ip 00007f6826353875 sp 00007ffdd36f9a40 error 4 in ld-2.28.so[7f6826332000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 231803:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9fdfeac4d800: inode [0x240000404:0x1866:0x0] mdc close failed: rc = -13
14[233307]: segfault at 0 ip 0000558f89c62b47 sp 00007ffe2857df70 error 6 in 14[558f89c5e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[245218]: segfault at 8 ip 00007fdd40444875 sp 00007ffcfab37fb0 error 4 in ld-2.28.so[7fdd40423000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[250385]: segfault at 8 ip 00007fd632ff9875 sp 00007ffd51563ad0 error 4 in ld-2.28.so[7fd632fd8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ODEBUG: object 000000009d936345 is on stack 000000003e550f2a, but NOT annotated.
WARNING: CPU: 3 PID: 7416 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 3 PID: 7416 Comm: mdt00_000 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: de 88 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffab8bc6f8f668 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffab8bc6f8f7a0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9fe1b22de5a8 RDI: ffff9fe1b22de5a8
RBP: ffffab8bc6f8f6b8 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffab8bc6f8f438 R12: ffffffff8a88f348
R13: ffffffff89505aa0 R14: 0000000000009f80 R15: ffff9fe001bd9c08
FS: 0000000000000000(0000) GS:ffff9fe1b22c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000562da2b6c000 CR3: 00000001d260a000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1164/0x2560 [lod]
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_declare_instantiate_components+0xc9/0x2b0 [lod]
lod_declare_update_plain+0x874/0xc40 [lod]
lod_declare_layout_change+0x742/0xe00 [lod]
mdd_declare_layout_change+0x63/0x170 [mdd]
mdd_layout_change+0xc8d/0x1580 [mdd]
mdt_layout_change+0x4df/0x6d0 [mdt]
? mdt_intent_fixup_resent+0x41/0x2e0 [mdt]
mdt_intent_layout+0xaaa/0x1540 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_open+0x5c0/0x5c0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 7b7f189b6aa33092 ]---
16[258633]: segfault at 55ce5f9d2000 ip 000055ce5f9d2000 sp 00007ffec83f2180 error 14 in 16[55ce5fbd2000+1000]
Code: Unable to access opcode bytes at RIP 0x55ce5f9d1fd6.
16[283391]: segfault at 557437b98000 ip 0000557437b98000 sp 00007ffd5e876bc8 error 14 in 16[557437d98000+1000]
Code: Unable to access opcode bytes at RIP 0x557437b97fd6.
6[287482]: segfault at 700e4707 ip 0000562f86f37560 sp 00007ffd8c3ea458 error 4 in 6[562f86f31000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 0e <38> 83 07 47 0e 70 68 0e 78 51 0e 80 01 43 0e 88 01 42 0e 90 01 5a
15[288702]: segfault at 8 ip 00007f76167a4875 sp 00007ffdfabcf410 error 4 in ld-2.28.so[7f7616783000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[291683]: segfault at 8 ip 00007f19b0a6f875 sp 00007ffeca287ec0 error 4 in ld-2.28.so[7f19b0a4e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[292604]: segfault at 8 ip 00007f0421544875 sp 00007ffcb1b3e1b0 error 4 in ld-2.28.so[7f0421523000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[293379]: segfault at 8 ip 00007fc46613e875 sp 00007ffcaa4b3ce0 error 4 in ld-2.28.so[7fc46611d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 31789:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x1f0a:0x0] ACL: rc = -2

LustreError: 63183:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 63183:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 9 PID: 63183 Comm: ll_sa_63081 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 8959:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 561, rollback = 7
Lustre: 8959:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8959:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8959:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/561/0, punch: 0/0/0, quota 1/3/0
Lustre: 8959:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8959:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6097:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 525, rollback = 7
Lustre: 6097:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6097:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6097:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6097:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6097:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6097:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/525/0, punch: 0/0/0, quota 1/3/0
Lustre: 6097:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6097:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6097:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6097:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6097:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8959:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8959:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8959:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8959:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8959:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8959:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8959:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8959:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8959:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8959:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8959:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8959:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 1 previous similar message
19[17677]: segfault at 55c041e8c000 ip 000055c041e8c000 sp 00007ffe58dc5ba8 error 14 in 19[55c04208c000+1000]
Code: Unable to access opcode bytes at RIP 0x55c041e8bfd6.
traps: 4[19697] trap invalid opcode ip:555a4500347a sp:7ffd6c32f8b8 error:0 in 4[555a44ffe000+7000]
Lustre: 8886:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8886:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8886:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8886:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8886:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8886:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8886:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8886:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8886:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8886:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8886:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8886:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6096:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 591, rollback = 7
Lustre: 6096:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 6096:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6096:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6096:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6096:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6096:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/591/0, punch: 0/0/0, quota 4/150/0
Lustre: 6096:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6096:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6096:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6096:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6096:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 5819:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9f40f7f0bb80 x1878829467924352/t4294979243(0) o101->1171efdf-6ae7-4f29-9f0f-d138e54b6a14@0@lo:609/0 lens 376/840 e 0 to 0 dl 1791791524 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 6098:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6098:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 6098:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6098:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6098:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6098:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6098:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6098:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6098:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6098:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 6098:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6098:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: ll_ost_io00_004: service thread pid 8959 was inactive for 41.057 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_004 state:R running task stack:0 pid:8959 ppid:2 flags:0x80004080
Call Trace:
? ldiskfs_es_lookup_extent+0xd7/0x350 [ldiskfs]
ldiskfs_map_blocks+0x65/0x920 [ldiskfs]
osd_fallocate_preallocate.isra.36+0x396/0xc30 [osd_ldiskfs]
? osd_declare_attr_qid+0x276/0x390 [osd_ldiskfs]
? osd_declare_attr_set+0x560/0x7c0 [osd_ldiskfs]
? ofd_object_fallocate+0x3e9/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 8886:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8886:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 8886:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8886:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8886:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8886:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8886:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8886:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8886:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8886:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8886:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8886:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 7 previous similar messages
8[38210]: segfault at 0 ip 00005585462de200 sp 00007fff8632e618 error 6 in 8[5585462dc000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 8886:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8886:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 11 previous similar messages
Lustre: 8886:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8886:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8886:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8886:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8886:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8886:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8886:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8886:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 11 previous similar messages
Lustre: 8886:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8886:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 11 previous similar messages
8[39503]: segfault at 561c9705a000 ip 0000561c9705a000 sp 00007fffe4de8be8 error 14 in 8[561c9725a000+1000]
Code: Unable to access opcode bytes at RIP 0x561c97059fd6.
LustreError: 5809:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000ee19d47a/0x1886bc14f71a615a lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x22:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->20479) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x1886bc14f71a614c expref: 5030 pid: 11032 timeout: 245 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffff9f40f32dd000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff9f40f32dd000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff9f40f32dd000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
LustreError: 40835:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff9f40f32dd000: namespace resource [0x300000400:0x22:0x0].0x0 (ffff9f4094b2c300) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff9f40f32dd000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_004: service thread pid 8959 completed after 103.740s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
10[41470]: segfault at 8 ip 00007fc3928ac875 sp 00007ffe9b03ee40 error 4 in ld-2.28.so[7fc39288b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 21947:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 21947:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 25 previous similar messages
Lustre: 21947:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 21947:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 21947:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 21947:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 21947:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 21947:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 21947:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 21947:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 21947:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21947:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 8872:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5834:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2

LustreError: 192385:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 192385:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 11 PID: 192385 Comm: ll_sa_192376 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 8294:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa09886b59500 x1878411751625728/t4294967400(0) o101->38dd45b9-e424-4e06-a59c-97dfb27152ec@0@lo:76/0 lens 376/840 e 0 to 0 dl 1791393106 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 5807:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5807:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 5807:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0
Lustre: 5807:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 1/3/0
Lustre: 5807:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 7/117/0, delete: 3/6/0
Lustre: 5807:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
17[10611]: segfault at 556e4e80a000 ip 0000556e4e80a000 sp 00007fff64ff3d58 error 14 in 17[556e4ea0a000+1000]
Code: Unable to access opcode bytes at RIP 0x556e4e809fd6.
17[10577]: segfault at 55c18fe48000 ip 000055c18fe48000 sp 00007ffe96939638 error 14 in 17[55c190048000+1000]
Code: Unable to access opcode bytes at RIP 0x55c18fe47fd6.
Lustre: 9072:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 9072:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 9072:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 9072:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 9072:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 9072:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
6[12996]: segfault at 8 ip 00007fee9cfa4875 sp 00007ffe59cf7b50 error 4 in ld-2.28.so[7fee9cf83000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 5807:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 5807:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 5807:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 5807:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0
Lustre: 5807:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 5807:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 1/3/0
Lustre: 5807:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 5807:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 7/117/0, delete: 3/6/1
Lustre: 5807:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 5807:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 5807:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 1 previous similar message
0[13793]: segfault at 0 ip 000055e49f423b47 sp 00007ffe79efe110 error 6 in 0[55e49f41f000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 6066:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6066:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6066:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6066:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6066:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6066:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6066:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6066:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6066:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6066:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6066:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6066:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6067:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6067:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6067:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6067:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6067:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 11646:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11646:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11646:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 11646:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11646:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11646:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11646:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11646:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11646:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11646:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 11646:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 11646:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages
1[19118]: segfault at 8 ip 00007f71466d8875 sp 00007ffc993c6830 error 4 in ld-2.28.so[7f71466b7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6067:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6067:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6067:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6067:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6067:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 23371:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6067:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 4 previous similar messages
Lustre: 6067:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 23371:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 4 previous similar messages
Lustre: 6067:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 5807:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5807:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: ll_ost_io00_003: service thread pid 9072 was inactive for 43.079 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_003 state:D stack:0 pid:9072 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x99a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 27302:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 27302:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 27302:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27302:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 27302:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27302:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 27302:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 27302:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 27302:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27302:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 6 previous similar messages
Lustre: 27302:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 27302:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 5 previous similar messages
9[34690]: segfault at 8 ip 00007f046a40d875 sp 00007ffc0de3a8e0 error 4 in ld-2.28.so[7f046a3ec000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6068:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6068:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 27302:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27302:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 27302:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27302:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 27302:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/591/0, punch: 0/0/0, quota 4/150/0
Lustre: 27302:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 27302:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27302:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 27302:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 27302:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 19 previous similar messages
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 0000000032ee934a/0x13276b02bcd1facc lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x4f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 16384->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bcd1fa94 expref: 71799 pid: 24114 timeout: 327 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffffa098e39c5800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffffa098e39c5800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffffa098e39c5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3209:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x59a:0x0]// may get corrupted (rc -108)
Lustre: 3206:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x539:0x0]// may get corrupted (rc -108)
Lustre: 3208:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x557:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffffa098e39c5800: Connection restored to 0@lo (at 0@lo)
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6067:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 23 previous similar messages
INFO: task fallocate:18216 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:18216 ppid:7733 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f9ce0e6970b
Code: Unable to access opcode bytes at RIP 0x7f9ce0e696e1.
RSP: 002b:00007ffd41af7128 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9ce0e6970b
RDX: 0000000000002d91 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000075e3 R11: 0000000000000246 R12: 00007ffd41af7348
R13: 0000000000000000 R14: 00007ffd41af7188 R15: 0000000000002d91
INFO: task fallocate:29832 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:29832 ppid:7649 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fd69c23770b
Code: Unable to access opcode bytes at RIP 0x7fd69c2376e1.
RSP: 002b:00007ffc32d62618 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fd69c23770b
RDX: 00000000000026f9 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000010af R11: 0000000000000246 R12: 00007ffc32d62838
R13: 0000000000000000 R14: 00007ffc32d62678 R15: 00000000000026f9
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000008fd80e7/0x13276b02bcd63c80 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x5e:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8192->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bcd63c64 expref: 1439 pid: 6953 timeout: 372 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffffa09901f67800: operation ost_sync to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffffa09901f67800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffffa09901f67800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3219:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x6eb:0x0]/ may get corrupted (rc -5)
LustreError: 55263:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffffa09901f67800: namespace resource [0x280000400:0x65:0x0].0x0 (ffffa09898d7f500) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffffa09901f67800: Connection restored to 0@lo (at 0@lo)
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000c4cf0c88/0x13276b02bcd7bb81 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x65:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 24576->36863) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bcd7baaf expref: 1677 pid: 37168 timeout: 384 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffffa098e39c5800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffffa098e39c5800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffffa098e39c5800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: 56630:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffffa098e39c5800: namespace resource [0x280000400:0x65:0x0].0x0 (ffffa099023bc700) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 56630:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-ffffa098e39c5800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_003: service thread pid 9072 completed after 227.998s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 60273:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffffa09901f67800: inode [0x200000402:0x8ee:0x0] mdc close failed: rc = -13
Lustre: 5807:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 5807:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 27302:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 27302:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 14 previous similar messages
Lustre: 27302:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 27302:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 14 previous similar messages
Lustre: 27302:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 27302:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 14 previous similar messages
Lustre: 27302:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 27302:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 14 previous similar messages
Lustre: 27302:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 27302:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 14 previous similar messages
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 105s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 000000001338c8a5/0x13276b02bce1bdf1 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x94:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 12288->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bce1bddc expref: 6031 pid: 6953 timeout: 467 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffffa098e39c5800: operation ost_fallocate to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffffa098e39c5800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffffa098e39c5800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3209:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0xc39:0x0]/ may get corrupted (rc -108)
Lustre: 3208:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0xc5b:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffffa098e39c5800: Connection restored to 0@lo (at 0@lo)
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000005d7abe55/0x13276b02bce3472c lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x98:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 0->20479) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x13276b02bce34725 expref: 2279 pid: 12086 timeout: 481 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffffa09901f67800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffffa09901f67800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffffa09901f67800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3213:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0xd2e:0x0]/ may get corrupted (rc -108)
LustreError: 73302:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffffa09901f67800: namespace resource [0x280000400:0x98:0x0].0x0 (ffffa09899103100) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffffa09901f67800: Connection restored to 0@lo (at 0@lo)
Lustre: 38108:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 38108:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 45 previous similar messages
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000f2790e7d/0x13276b02bcf14be0 lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x124:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8192->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bcf14bd9 expref: 6082 pid: 6060 timeout: 585 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0003-osc-ffffa098e39c5800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffffa098e39c5800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffffa098e39c5800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3209:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x13ef:0x0]/ may get corrupted (rc -108)
Lustre: 3206:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x13ad:0x0]/ may get corrupted (rc -108)
LustreError: 92455:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffffa098e39c5800: namespace resource [0x2c0000400:0x124:0x0].0x0 (ffffa09924cac100) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 92455:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffffa098e39c5800: Connection restored to 0@lo (at 0@lo)
INFO: task fallocate:64751 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:64751 ppid:7694 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7cb7e5f70b
Code: Unable to access opcode bytes at RIP 0x7f7cb7e5f6e1.
RSP: 002b:00007fff64ee2a48 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7cb7e5f70b
RDX: 000000000000584b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005d1f R11: 0000000000000246 R12: 00007fff64ee2c68
R13: 0000000000000000 R14: 00007fff64ee2aa8 R15: 000000000000584b
INFO: task chmod:65035 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:65035 ppid:7679 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbfa227d258
Code: Unable to access opcode bytes at RIP 0x7fbfa227d22e.
RSP: 002b:00007ffd7b521478 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbfa227d258
RDX: 0000000000000120 RSI: 000055dda54a8500 RDI: 00000000ffffff9c
RBP: 000055dda54a8470 R08: 0000000000000000 R09: 0000000000000120
R10: 0000000000000fff R11: 0000000000000246 R12: 000055dda4332bb4
R13: 000055dda54a8500 R14: 000055dda54a9730 R15: 0000000000008000
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000d28082a3/0x13276b02bcf688a6 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x150:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 20480->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bcf68898 expref: 16 pid: 8694 timeout: 625 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffffa098e39c5800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffffa098e39c5800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffffa098e39c5800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3212:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x1698:0x0]/ may get corrupted (rc -108)
Lustre: 3210:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x1637:0x0]/ may get corrupted (rc -108)
Lustre: 3210:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x165f:0x0]/ may get corrupted (rc -108)
LustreError: 100585:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffffa098e39c5800: namespace resource [0x280000400:0x150:0x0].0x0 (ffffa098fe15f800) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffffa098e39c5800: Connection restored to 0@lo (at 0@lo)
19[106887]: segfault at 8 ip 00007f78087e4875 sp 00007ffd03482b80 error 4 in ld-2.28.so[7f78087c3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[110939]: segfault at 8 ip 00007ff81fd39875 sp 00007ffea787caa0 error 4 in ld-2.28.so[7ff81fd18000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 38997:0:(osd_handler.c:2099:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 38997:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 73 previous similar messages
Lustre: 38997:0:(osd_handler.c:2106:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 38997:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 73 previous similar messages
Lustre: 38997:0:(osd_handler.c:2116:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 38997:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 73 previous similar messages
Lustre: 38997:0:(osd_handler.c:2123:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 38997:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 73 previous similar messages
Lustre: 38997:0:(osd_handler.c:2130:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 38997:0:(osd_handler.c:2130:osd_trans_dump_creds()) Skipped 73 previous similar messages
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000ea4cda55/0x13276b02bd05fe5b lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x1b3:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 4096->16383) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bd05fe07 expref: 18 pid: 24114 timeout: 731 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffffa09901f67800: operation ost_punch to node 0@lo failed: rc = -107
LustreError: 8334:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa099234d7000 x1878411830819584/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: lustre-OST0001-osc-ffffa09901f67800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffffa09901f67800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3213:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x1dea:0x0]/ may get corrupted (rc -108)
Lustre: 3210:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x1f69:0x0]// may get corrupted (rc -108)
Lustre: 3213:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x1fee:0x0]/ may get corrupted (rc -108)
Lustre: 3215:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x1fff:0x0]/ may get corrupted (rc -108)
LustreError: 123173:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffffa09901f67800: namespace resource [0x280000400:0x232:0x0].0x0 (ffffa099285ffb00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffffa09901f67800: Connection restored to 0@lo (at 0@lo)
INFO: task fallocate:64751 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:64751 ppid:7694 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7cb7e5f70b
Code: Unable to access opcode bytes at RIP 0x7f7cb7e5f6e1.
RSP: 002b:00007fff64ee2a48 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7cb7e5f70b
RDX: 000000000000584b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005d1f R11: 0000000000000246 R12: 00007fff64ee2c68
R13: 0000000000000000 R14: 00007fff64ee2aa8 R15: 000000000000584b
INFO: task chmod:65035 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:65035 ppid:7679 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbfa227d258
Code: Unable to access opcode bytes at RIP 0x7fbfa227d22e.
RSP: 002b:00007ffd7b521478 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbfa227d258
RDX: 0000000000000120 RSI: 000055dda54a8500 RDI: 00000000ffffff9c
RBP: 000055dda54a8470 R08: 0000000000000000 R09: 0000000000000120
R10: 0000000000000fff R11: 0000000000000246 R12: 000055dda4332bb4
R13: 000055dda54a8500 R14: 000055dda54a9730 R15: 0000000000008000
Lustre: 38997:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 38997:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 53 previous similar messages
9[133207]: segfault at 8 ip 00007fd2f48e7875 sp 00007ffcbc670170 error 4 in ld-2.28.so[7fd2f48c6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[137317]: segfault at 0 ip 00005640f0f38b47 sp 00007fffac60c940 error 6 in 11[5640f0f34000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 12118:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: ll_ost_io00_004: service thread pid 11646 was inactive for 475.338 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_004 state:R running task stack:0 pid:11646 ppid:2 flags:0x80004080
Call Trace:
? _raw_read_unlock+0x16/0x30
? osd_tx_was_declared+0xd5/0x3a0 [osd_ldiskfs]
? osd_trunc_unlock_all+0x78/0x1e0 [osd_ldiskfs]
? osd_trans_stop+0x4ea/0xe10 [osd_ldiskfs]
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x34c/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
INFO: task fallocate:64751 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:64751 ppid:7694 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7cb7e5f70b
Code: Unable to access opcode bytes at RIP 0x7f7cb7e5f6e1.
RSP: 002b:00007fff64ee2a48 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7cb7e5f70b
RDX: 000000000000584b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005d1f R11: 0000000000000246 R12: 00007fff64ee2c68
R13: 0000000000000000 R14: 00007fff64ee2aa8 R15: 000000000000584b
INFO: task chmod:65035 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:65035 ppid:7679 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbfa227d258
Code: Unable to access opcode bytes at RIP 0x7fbfa227d22e.
RSP: 002b:00007ffd7b521478 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbfa227d258
RDX: 0000000000000120 RSI: 000055dda54a8500 RDI: 00000000ffffff9c
RBP: 000055dda54a8470 R08: 0000000000000000 R09: 0000000000000120
R10: 0000000000000fff R11: 0000000000000246 R12: 000055dda4332bb4
R13: 000055dda54a8500 R14: 000055dda54a9730 R15: 0000000000008000
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000c45ff00b/0x13276b02bd1df7d3 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x28b:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 0->16383) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bd1df7cc expref: 28960 pid: 61412 timeout: 888 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 132349:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffffa098fe9e5400 x1878411852823040/t0(0) o105->lustre-OST0002@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffffa09901f67800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffffa09901f67800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffffa09901f67800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3218:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x285b:0x0]// may get corrupted (rc -5)
Lustre: 3214:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x275a:0x0]/ may get corrupted (rc -108)
Lustre: 3217:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x29f6:0x0]/ may get corrupted (rc -108)
Lustre: 3213:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x27ca:0x0]/ may get corrupted (rc -108)
Lustre: 3204:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x2a19:0x0]/ may get corrupted (rc -108)
Lustre: 3219:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x29cb:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffffa09901f67800: Connection restored to 0@lo (at 0@lo)
10[155543]: segfault at 0 ip 000055b83e21fb47 sp 00007ffdcbd7b7c0 error 6 in 10[55b83e21b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: ll_ost_io00_000: service thread pid 6066 was inactive for 488.621 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_000 state:D stack:0 pid:6066 ppid:2 flags:0x80004080
Call Trace:
? __change_page_attr_set_clr+0x75/0x3b0
? __alloc_pages_nodemask+0x22a/0x550
? __find_get_block+0x118/0x540
? __might_sleep+0x60/0xc0
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? ldiskfs_do_update_inode+0x79a/0xbb0 [ldiskfs]
? jbd2_journal_get_write_access+0x49/0xf0
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? __find_get_block+0x118/0x540
? debug_check_no_obj_freed+0x16f/0x2e5
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? slab_free_freelist_hook.isra.49+0x111/0x230
? start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
? jbd2__journal_start+0x11f/0x300
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
? dt_trans_start+0x20/0x70 [obdclass]
? ofd_trans_start+0x7d/0xf0 [ofd]
? ofd_object_fallocate+0x6cf/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
LustreError: 3203:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffffa098e39c5800: granted 8437760 but already consumed 9367552
LustreError: lustre-OST0000-osc-ffffa098e39c5800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3216:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x2adc:0x0]/ may get corrupted (rc -108)
Lustre: 3218:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x2aee:0x0]/ may get corrupted (rc -108)
Lustre: 83526:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 83526:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
3[164543]: segfault at 8 ip 00007fbc7c2c5875 sp 00007ffd3dc4f540 error 4 in ld-2.28.so[7fbc7c2a4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[167345]: segfault at 8 ip 00007f43a4a7c875 sp 00007ffe34589d00 error 4 in ld-2.28.so[7f43a4a5b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:64751 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:64751 ppid:7694 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7cb7e5f70b
Code: Unable to access opcode bytes at RIP 0x7f7cb7e5f6e1.
RSP: 002b:00007fff64ee2a48 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f7cb7e5f70b
RDX: 000000000000584b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005d1f R11: 0000000000000246 R12: 00007fff64ee2c68
R13: 0000000000000000 R14: 00007fff64ee2aa8 R15: 000000000000584b
INFO: task chmod:65035 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:65035 ppid:7679 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbfa227d258
Code: Unable to access opcode bytes at RIP 0x7fbfa227d22e.
RSP: 002b:00007ffd7b521478 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fbfa227d258
RDX: 0000000000000120 RSI: 000055dda54a8500 RDI: 00000000ffffff9c
RBP: 000055dda54a8470 R08: 0000000000000000 R09: 0000000000000120
R10: 0000000000000fff R11: 0000000000000246 R12: 000055dda4332bb4
R13: 000055dda54a8500 R14: 000055dda54a9730 R15: 0000000000008000
Lustre: 9072:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffffa09920a6d080 x1878411789268096/t0(0) o22->38dd45b9-e424-4e06-a59c-97dfb27152ec@0@lo:246/0 lens 440/432 e 12 to 0 dl 1791394031 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000071bf362f/0x13276b02bd35423d lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x367:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 16384->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x13276b02bd35422f expref: 669 pid: 61412 timeout: 1047 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 5782:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
LustreError: lustre-OST0003-osc-ffffa098e39c5800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffffa098e39c5800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0003-osc-ffffa098e39c5800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3212:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x33ed:0x0]/ may get corrupted (rc -108)
Lustre: 3214:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000401:0x3294:0x0]/ may get corrupted (rc -108)
Lustre: 3217:0:(llite_lib.c:4336:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.102@tcp:/lustre/fid: [0x200000402:0x341e:0x0]/ may get corrupted (rc -108)
LustreError: 183602:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffffa098e39c5800: namespace resource [0x2c0000400:0x367:0x0].0x0 (ffffa0992316f000) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 183602:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffffa098e39c5800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
15[186123]: segfault at 8 ip 00007f85816e8875 sp 00007fffe64107b0 error 4 in ld-2.28.so[7f85816c7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48

LustreError: 436026:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 436026:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 10 PID: 436026 Comm: ll_sa_435462 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30

Lustre: 9883:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff923dd748e900 x1877479558052992/t4294967573(0) o101->fa7905ca-1a9d-489b-b0a3-bd3f01ccf1c5@0@lo:473/0 lens 376/864 e 0 to 0 dl 1790504113 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 7542:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7542:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7542:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7542:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7542:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7542:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11020:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff923e48e70a80 x1877479559013376/t4294968401(0) o101->fa7905ca-1a9d-489b-b0a3-bd3f01ccf1c5@0@lo:479/0 lens 376/816 e 0 to 0 dl 1790504119 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 12999:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x240000404:0x2c:0x0] mdc close failed: rc = -13
Lustre: 11704:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 11704:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 11704:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11704:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11704:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 11704:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11704:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11704:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11704:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11704:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11704:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11704:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
traps: 18[13868] trap invalid opcode ip:563d979ce84c sp:7ffda3e97cc0 error:0 in 18[563d979cc000+7000]
LustreError: 14944:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0ec59800: inode [0x200000403:0x4a:0x0] mdc close failed: rc = -13
Lustre: 11060:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff923e073fe200 x1877479560702208/t4294969103(0) o101->fa7905ca-1a9d-489b-b0a3-bd3f01ccf1c5@0@lo:490/0 lens 376/840 e 0 to 0 dl 1790504130 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 14287:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 14287:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14287:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14287:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 14287:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14287:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14287:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
9[17840]: segfault at 0 ip 000055c26c66ab47 sp 00007ffefe5b14e0 error 6 in 9[55c26c666000+7000]
Code: Unable to access opcode bytes at RIP 0x55c26c66ab1d.
Lustre: 12109:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12109:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 12109:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12109:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12109:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12109:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12109:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 12109:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12109:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12109:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12109:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12109:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
19[18952]: segfault at 8 ip 00007f30aa0b6875 sp 00007ffd3bdc2ab0 error 4 in ld-2.28.so[7f30aa095000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 14287:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 14287:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14287:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 14287:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 14287:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14287:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14287:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14287:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 12109:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12109:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 12109:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12109:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12109:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12109:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12109:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 12109:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12109:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12109:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12109:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12109:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: ll_ost_io00_001: service thread pid 7542 was inactive for 40.242 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_002 state:R
Lustre: Skipped 1 previous similar message
task:ll_ost_io00_001 state:R running task stack:0 pid:7542 ppid:2 flags:0x80004080
Call Trace:
? 0xffffffffa9a00000
? __change_page_attr_set_clr+0x75/0x3b0
? __alloc_pages_nodemask+0x22a/0x550
? __find_get_block+0x118/0x540
? ldiskfs_es_lookup_extent+0xd7/0x350 [ldiskfs]
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
? __getblk_gfp+0x59/0x5b0
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? __ldiskfs_journal_get_write_access+0x64/0xd0 [ldiskfs]
? osd_ldiskfs_write+0x77b/0x830 [osd_ldiskfs]
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? do_raw_spin_unlock+0x79/0x190
? debug_check_no_obj_freed+0x16f/0x2e5
? slab_free_freelist_hook.isra.49+0x111/0x230
? osd_trunc_unlock_all+0xf2/0x1e0 [osd_ldiskfs]
? kfree+0x28a/0x380
? __check_object_size+0x10e/0x26d
? __probe_kernel_read+0x97/0x100
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? kmem_cache_alloc_trace+0x30f/0x440
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? ofd_trans_create+0x7f/0x2b0 [ofd]
? ofd_object_fallocate+0x36b/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
stack:0 pid:7543 ppid:2 flags:0x80004080
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 515 < left 520, rollback = 2
Lustre: ll_ost_io00_005: service thread pid 11704 was inactive for 42.456 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_005 state:R running task stack:0 pid:11704 ppid:2 flags:0x80004080
Call Trace:
? osd_fallocate_preallocate.isra.37+0x452/0xc30 [osd_ldiskfs]
? ldiskfs_mark_iloc_dirty+0x4a/0x120 [ldiskfs]
ldiskfs_mark_inode_dirty+0xd8/0x390 [ldiskfs]
osd_fallocate_preallocate.isra.37+0x452/0xc30 [osd_ldiskfs]
? ofd_trans_stop+0x32/0x90 [ofd]
? ofd_object_fallocate+0x99a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 15860:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 15860:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 15860:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15860:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 10 previous similar messages
Lustre: 15860:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 15860:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 10 previous similar messages
Lustre: 15860:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 15860:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 10 previous similar messages
Lustre: 15860:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15860:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 10 previous similar messages
Lustre: 15860:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15860:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 10 previous similar messages
9[27306]: segfault at 0 ip 0000564f3851db47 sp 00007fff078d4020 error 6 in 9 (deleted)[564f38519000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 28711:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x240000403:0x1da:0x0] mdc close failed: rc = -13
Lustre: 6761:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 516 < left 520, rollback = 2
Lustre: 18671:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 18671:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 13 previous similar messages
Lustre: 18671:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18671:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 18671:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18671:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 18671:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 18671:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 18671:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18671:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 18671:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18671:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 000000009f0ad71c/0x153d9cde729eb1aa lrc: 3/0,0 mode: PW/PW res: [0x340000402:0xc:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 28672->49151) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x153d9cde729eb13a expref: 930 pid: 8039 timeout: 226 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 16340:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e0e7e5080 x1877479578095232/t0(0) o106->lustre-OST0002@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff923e0ec59800: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff923e0ec59800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff923e0ec59800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2e1:0x0]/ may get corrupted (rc -5)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x308:0x0]// may get corrupted (rc -5)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2a3:0x0]/ may get corrupted (rc -5)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1cc:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x344:0x0]/ may get corrupted (rc -5)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x30c:0x0]/ may get corrupted (rc -5)
Lustre: lustre-OST0002-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_005: service thread pid 11704 completed after 106.964s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000be5160df/0x153d9cde729e772d lrc: 3/0,0 mode: PW/PW res: [0x380000402:0xc:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->36863) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde729e70c0 expref: 38 pid: 8812 timeout: 235 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: lustre-OST0003-osc-ffff923e0fed0800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff923e0fed0800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff923e0fed0800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2ce:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2d1:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x366:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x1eb:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2c9:0x0]// may get corrupted (rc -108)
LustreError: 39104:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff923e0fed0800: namespace resource [0x380000402:0xc:0x0].0x0 (ffff923d8ac75b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff923e0fed0800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_002: service thread pid 7543 completed after 113.624s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_001: service thread pid 7542 completed after 113.443s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
9[40783]: segfault at 8 ip 00007f0dc35d2875 sp 00007fffba765320 error 4 in ld-2.28.so[7f0dc35b1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 48657:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x200000404:0x532:0x0] mdc close failed: rc = -13
14[50147]: segfault at 8 ip 00007ff83565c875 sp 00007ffd02387650 error 4 in ld-2.28.so[7ff83563b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 00000000d14af8a0/0x153d9cde72a6e812 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0x2d:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 12288->16383) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde72a6e7f6 expref: 6649 pid: 16340 timeout: 278 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: lustre-OST0003-osc-ffff923e0ec59800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0003-osc-ffff923e0ec59800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff923e0ec59800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x26b:0x0]/ may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x42e:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x509:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x3dd:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x543:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x510:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x37b:0x0]// may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2f7:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0003-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
5[51457]: segfault at 8 ip 00007f1b95439875 sp 00007fff0aa293f0 error 4 in ld-2.28.so[7f1b95418000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 30270:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 30270:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 30270:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 30270:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 30270:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 30270:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 30270:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 30270:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 30270:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 30270:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 30270:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 30270:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 514 < left 520, rollback = 2
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
12[55596]: segfault at 8 ip 00007feb75053875 sp 00007ffddb7f3050 error 4 in ld-2.28.so[7feb75032000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 12474:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 000000009b0af289/0x153d9cde72b0baaf lrc: 3/0,0 mode: PW/PW res: [0x340000401:0x4b:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 24576->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde72b0baa8 expref: 30 pid: 18339 timeout: 332 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: lustre-OST0002-osc-ffff923e0ec59800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff923e0ec59800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff923e0ec59800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x5aa:0x0]// may get corrupted (rc -5)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x5a9:0x0]// may get corrupted (rc -5)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x594:0x0]/ may get corrupted (rc -5)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x58e:0x0]/ may get corrupted (rc -5)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x6f4:0x0]/ may get corrupted (rc -5)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x5a2:0x0]/ may get corrupted (rc -5)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x6aa:0x0]/ may get corrupted (rc -5)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x4c3:0x0]/ may get corrupted (rc -5)
Lustre: lustre-OST0002-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_007: service thread pid 14287 was inactive for 174.658 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
INFO: task fallocate:28730 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:28730 ppid:9340 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0a3d89970b
Code: Unable to access opcode bytes at RIP 0x7f0a3d8996e1.
RSP: 002b:00007ffd171b7608 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0a3d89970b
RDX: 00000000000069d4 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000007d8f R11: 0000000000000246 R12: 00007ffd171b7828
R13: 0000000000000001 R14: 00007ffd171b7668 R15: 00000000000069d4
INFO: task fallocate:39412 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:39412 ppid:9709 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fb863de270b
Code: Unable to access opcode bytes at RIP 0x7fb863de26e1.
RSP: 002b:00007ffcd282fc58 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb863de270b
RDX: 0000000000007f5a RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000e5f R11: 0000000000000246 R12: 00007ffcd282fe78
R13: 0000000000000001 R14: 00007ffcd282fcb8 R15: 0000000000007f5a
INFO: task fallocate:40930 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:40930 ppid:9542 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff9c72cf70b
Code: Unable to access opcode bytes at RIP 0x7ff9c72cf6e1.
RSP: 002b:00007ffe0dbd5d08 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff9c72cf70b
RDX: 00000000000019a5 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004e5d R11: 0000000000000246 R12: 00007ffe0dbd5f28
R13: 0000000000000000 R14: 00007ffe0dbd5d68 R15: 00000000000019a5
17[76905]: segfault at 8 ip 00007f2c7d763875 sp 00007ffd97a0e600 error 4 in ld-2.28.so[7f2c7d742000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 0000000037e4d255/0x153d9cde72bac752 lrc: 3/0,0 mode: PW/PW res: [0x2c0000401:0x68:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 28672->49151) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde72bac713 expref: 2444 pid: 16340 timeout: 379 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: lustre-OST0000-osc-ffff923e0ec59800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 9833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790504366 with bad export cookie 1530551927595349218
Lustre: lustre-OST0000-osc-ffff923e0ec59800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff923e0ec59800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x6f2:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x6a6:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x632:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x6d4:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x761:0x0]// may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x642:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
19[80867]: segfault at 8 ip 00007f3dc880e875 sp 00007ffd617db6d0 error 4 in ld-2.28.so[7f3dc87ed000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000177252d9/0x153d9cde72bd57a7 lrc: 3/0,0 mode: PW/PW res: [0x340000401:0x6c:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 28672->45055) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde72bd5730 expref: 897 pid: 20211 timeout: 398 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 6734:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790504384 with bad export cookie 1530551927595348980
Lustre: lustre-OST0002-osc-ffff923e0fed0800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 8814:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e1bf58700 x1877479616689664/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff923e0fed0800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0xa8c:0x0]/ may get corrupted (rc -5)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x731:0x0]/ may get corrupted (rc -5)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x69d:0x0]/ may get corrupted (rc -5)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x83d:0x0]/ may get corrupted (rc -5)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x76f:0x0]/ may get corrupted (rc -5)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x831:0x0]// may get corrupted (rc -5)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x880:0x0]// may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x742:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x6a6:0x0]// may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x81a:0x0]// may get corrupted (rc -5)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0xaac:0x0]// may get corrupted (rc -5)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0xa5d:0x0]/ may get corrupted (rc -5)
Lustre: lustre-OST0002-osc-ffff923e0fed0800: Connection restored to 0@lo (at 0@lo)
Lustre: 6760:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 516 < left 520, rollback = 2
Lustre: 14298:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 14298:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 85 previous similar messages
Lustre: 14298:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14298:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 14298:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14298:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 14298:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 14298:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 14298:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14298:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 14298:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14298:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 90 previous similar messages
13[93968]: segfault at 8 ip 00007f9be6518875 sp 00007fff1702df70 error 4 in ld-2.28.so[7f9be64f7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 95120:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x280000404:0x7ee:0x0] mdc close failed: rc = -13
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000097357280/0x153d9cde72c81b2a lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x8c:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 12288->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde72c81a90 expref: 346 pid: 7537 timeout: 441 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 6736:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e35511180 x1877479626164736/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0003-osc-ffff923e0ec59800: operation ost_punch to node 0@lo failed: rc = -107
LustreError: 6736:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 3 previous similar messages
Lustre: lustre-OST0003-osc-ffff923e0ec59800: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff923e0ec59800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0xc49:0x0]// may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0xc1a:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0xc15:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0xbca:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x8e0:0x0]// may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0xc4e:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x8f0:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0xc41:0x0]/ may get corrupted (rc -108)
LustreError: 95906:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff923e0ec59800: namespace resource [0x380000401:0x8c:0x0].0x0 (ffff923ea0733100) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 95906:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
LustreError: 97977:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0ec59800: inode [0x280000403:0x9b8:0x0] mdc close failed: rc = -13
4[101197]: segfault at 2 ip 000055da1c53f6ea sp 00007fff94a4a898 error 6 in 4[55da1c53e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 0f 1f 44 00 00 f3 0f 1e fa f2 ff 25 ed 67 20 00 0f 1f 44 00 00
0[101837]: segfault at 8 ip 00007f8103fee875 sp 00007ffeaca53a40 error 4 in ld-2.28.so[7f8103fcd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:28730 blocked for more than 120 seconds.
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000a75285e5/0x153d9cde72d30772 lrc: 3/0,0 mode: PW/PW res: [0x300000402:0xff:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 0->32767) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x153d9cde72d306fb expref: 31 pid: 55651 timeout: 490 lvb_type: 0 lru_score: 0 lru_type: 0
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:28730 ppid:9340 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0a3d89970b
Code: Unable to access opcode bytes at RIP 0x7f0a3d8996e1.
RSP: 002b:00007ffd171b7608 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f0a3d89970b
RDX: 00000000000069d4 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000007d8f R11: 0000000000000246 R12: 00007ffd171b7828
R13: 0000000000000001 R14: 00007ffd171b7668 R15: 00000000000069d4
INFO: task fallocate:40930 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:40930 ppid:9542 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff9c72cf70b
Code: Unable to access opcode bytes at RIP 0x7ff9c72cf6e1.
RSP: 002b:00007ffe0dbd5d08 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007ff9c72cf70b
RDX: 00000000000019a5 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004e5d R11: 0000000000000246 R12: 00007ffe0dbd5f28
R13: 0000000000000000 R14: 00007ffe0dbd5d68 R15: 00000000000019a5
INFO: task dd:41280 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dd state:D stack:0 pid:41280 ppid:9461 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
vvp_io_setattr_start+0x4b5/0x5f0 [lustre]
? init_wait_var_entry+0x50/0x50
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? inode_newsize_ok+0x4f/0xa0
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
do_truncate+0x9a/0x130
do_last+0x8b6/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f4ac3bff332
LustreError: lustre-OST0001-osc-ffff923e0ec59800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Code: Unable to access opcode bytes at RIP 0x7f4ac3bff308.
RSP: 002b:00007fffdbbb6810 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055c347749120 RCX: 00007f4ac3bff332
RDX: 0000000000000241 RSI: 00007fffdbbb8c30 RDI: 00000000ffffff9c
RBP: 0000000000000001 R08: 00007fffdbbb8c5c R09: 0000000000000000
R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000241
R13: 00007fffdbbb8c30 R14: 0000000000000001 R15: 00007fffdbbb6af8
Lustre: lustre-OST0001-osc-ffff923e0ec59800: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff923e0ec59800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
INFO: task fallocate:50996 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:50996 ppid:9524 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7faf8921370b
Code: Unable to access opcode bytes at RIP 0x7faf892136e1.
RSP: 002b:00007ffc575dae08 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007faf8921370b
RDX: 00000000000023af RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000006e9d R11: 0000000000000246 R12: 00007ffc575db028
R13: 0000000000000000 R14: 00007ffc575dae68 R15: 00000000000023af
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0xde6:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x9a7:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xabe:0x0]// may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xa8c:0x0]/ may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xa02:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xaa5:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xa52:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
7[109237]: segfault at 8 ip 00007f4fddc3e875 sp 00007ffcb7eecb90 error 4 in ld-2.28.so[7f4fddc1d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: lustre-OST0003-osc-ffff923e0fed0800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: lustre-OST0003-osc-ffff923e0fed0800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0xbe8:0x0]// may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xce6:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x103e:0x0]// may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1083:0x0]// may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xcc5:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0xbd5:0x0]// may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1121:0x0]/ may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1153:0x0]// may get corrupted (rc -108)
LustreError: 123929:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff923e0fed0800: namespace resource [0x380000402:0x194:0x0].0x0 (ffff923e63df8700) refcount nonzero (1) after lock cleanup; forcing cleanup.
3[123381]: segfault at 8 ip 00007f5b6abee875 sp 00007ffc5dd48ca0 error 4 in ld-2.28.so[7f5b6abcd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[124831]: segfault at 0 ip 00005564dd57e200 sp 00007ffc21464278 error 6 in 9[5564dd57c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9[126655]: segfault at 55d469361f24 ip 000055d469361f24 sp 00007fffc3ced128 error 7 in 9[55d46935d000+7000]
Code: 84 00 00 00 00 00 f3 0f 1e fa c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 8b 15 45 2c 20 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ptlrpc_watchdog_fire: 1 callbacks suppressed
Lustre: ll_ost_io00_023: service thread pid 35050 was inactive for 279.656 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_023 state:D stack:0 pid:35050 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 00000000d27cba21/0x153d9cde72e483eb lrc: 3/0,0 mode: PW/PW res: [0x2c0000402:0x14f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 20480->53247) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x153d9cde72e483c1 expref: 1661 pid: 16304 timeout: 581 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
Lustre: lustre-OST0000-osc-ffff923e0fed0800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff923e0fed0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x11f7:0x0]// may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xdd4:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xcd3:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x12d4:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xe4c:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1272:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xe65:0x0]// may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xd1b:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff923e0fed0800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
Lustre: ll_ost_io00_023: service thread pid 35050 completed after 302.418s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 136518:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x240000404:0xde1:0x0] mdc close failed: rc = -13
LustreError: 17336:0:(tgt_grant.c:755:tgt_grant_check()) lustre-OST0001: cli 16a55e9a-6a6c-43a2-9a00-f43c107c6c8c claims 1990656 GRANT, real grant 0
LustreError: 17336:0:(ldlm_lib.c:3612:target_bulk_io()) @@@ bulk WRITE failed: rc = -107 req@ffff923e43495080 x1877479662155904/t0(0) o4->16a55e9a-6a6c-43a2-9a00-f43c107c6c8c@0@lo:703/0 lens 488/448 e 0 to 0 dl 1790505098 ref 1 fl Interpret:H/600/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: lustre-OST0001: Bulk IO write error with 16a55e9a-6a6c-43a2-9a00-f43c107c6c8c (at 0@lo), client will retry: rc = -107
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0001-osc-ffff923e0fed0800: granted 8437760 but already consumed 16367616
LustreError: lustre-OST0001-osc-ffff923e0fed0800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1348:0x0]// may get corrupted (rc -5)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xdd2:0x0]/ may get corrupted (rc -5)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0xcdb:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x12b0:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xe3a:0x0]// may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xeb7:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x133a:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x132e:0x0]// may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0xd1b:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x12a5:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x12c6:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xea4:0x0]// may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1297:0x0]// may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0xd5e:0x0]/ may get corrupted (rc -108)
Lustre: ll_ost_io00_007: service thread pid 14287 completed after 402.160s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 138757:0:(statahead.c:2416:start_statahead_thread()) lustre: invalid pattern 0X0.
7[140504]: segfault at 8 ip 00007f824491f875 sp 00007ffee9eba4c0 error 4 in ld-2.28.so[7f82448fe000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[142824]: segfault at 0 ip 000056069f129b47 sp 00007ffc7d2dd9b0 error 6 in 10[56069f125000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: lustre-OST0003-osc-ffff923e0ec59800: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0003-osc-ffff923e0ec59800: granted 8437760 but already consumed 15552512
LustreError: lustre-OST0003-osc-ffff923e0ec59800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x13a1:0x0]/ may get corrupted (rc -5)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xde3:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x12f7:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xe8d:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xe54:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xe4b:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xecf:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xebd:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x13cf:0x0]// may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1407:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xeb8:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xf03:0x0]// may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xf1d:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xe98:0x0]/ may get corrupted (rc -108)
LustreError: 143957:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff923e0ec59800: namespace resource [0x380000401:0x15e:0x0].0x0 (ffff923e3578db00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 143957:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 3 previous similar messages
19[145350]: segfault at 0 ip 000055e40d6590b0 sp 00007ffe5b75ca18 error 6 in 19[55e40d657000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
12[146972]: segfault at 562c23e93000 ip 0000562c23e93000 sp 00007ffe9302b728 error 14 in 12[562c24093000+1000]
Code: Unable to access opcode bytes at RIP 0x562c23e92fd6.
LustreError: 112143:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e3391c600 x1877479672184192/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0001-osc-ffff923e0ec59800: granted 8437760 but already consumed 9297920
LustreError: lustre-OST0001-osc-ffff923e0ec59800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: lustre-OST0002-osc-ffff923e0ec59800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xf17:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x14fe:0x0]// may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x14e2:0x0]// may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xf97:0x0]// may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x14a0:0x0]/ may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1497:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x14ed:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xf4b:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xf93:0x0]// may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xe64:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xee6:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0xdf3:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xf62:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xf08:0x0]// may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x14c0:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0xe6a:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x14d9:0x0]// may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xf3f:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1003:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x127a:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0xf72:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0xf95:0x0]// may get corrupted (rc -108)
Lustre: 20030:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 515 < left 520, rollback = 2
Lustre: 20030:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 133873:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 133873:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 249 previous similar messages
Lustre: 133873:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 133873:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 250 previous similar messages
Lustre: 133873:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 133873:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 250 previous similar messages
Lustre: 133873:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 133873:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 250 previous similar messages
Lustre: 133873:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 133873:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 250 previous similar messages
Lustre: 133873:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 133873:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 250 previous similar messages
18[168234]: segfault at 8 ip 00007f1fd0ace875 sp 00007ffcfb28b910 error 4 in ld-2.28.so[7f1fd0aad000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[168758]: segfault at 8 ip 00007fa40d1ba875 sp 00007ffe7b6d6850 error 4 in ld-2.28.so[7fa40d199000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[170978]: segfault at 8 ip 00007f3a87e4f875 sp 00007ffe07ee92e0 error 4 in ld-2.28.so[7f3a87e2e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:111790 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:111790 ppid:9407 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fb82a71a70b
Code: Unable to access opcode bytes at RIP 0x7fb82a71a6e1.
RSP: 002b:00007ffdca66aa48 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb82a71a70b
RDX: 0000000000006bd7 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000396b R11: 0000000000000246 R12: 00007ffdca66ac68
R13: 0000000000000000 R14: 00007ffdca66aaa8 R15: 0000000000006bd7
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000004c0d365b/0x153d9cde730d5c4e lrc: 3/0,0 mode: PW/PW res: [0x2c0000402:0x1ff:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 12288->32767) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde730d5c16 expref: 28 pid: 73052 timeout: 754 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 4 previous similar messages
LustreError: lustre-OST0000-osc-ffff923e0ec59800: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff923e0ec59800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 4 previous similar messages
LustreError: 19786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790504739 with bad export cookie 1530551927601882339
Lustre: Skipped 5 previous similar messages
LustreError: lustre-OST0001-osc-ffff923e0fed0800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
LustreError: lustre-OST0000-osc-ffff923e0ec59800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1991:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1778:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x129d:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x12a4:0x0]// may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x19f8:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x126b:0x0]/ may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x1358:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1a01:0x0]/ may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x12cc:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x19d7:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x19db:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1940:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1336:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x1372:0x0]/ may get corrupted (rc -108)
LustreError: 183885:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff923e0fed0800: namespace resource [0x300000401:0x18d:0x0].0x0 (ffff923ebc2d6d00) refcount nonzero (2) after lock cleanup; forcing cleanup.
LustreError: 183885:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x19ea:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 4 previous similar messages
7[185558]: segfault at 8 ip 00007fa957255875 sp 00007ffe7b66f4e0 error 4 in ld-2.28.so[7fa957234000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[198070]: segfault at 8 ip 00007f958e2e0875 sp 00007ffe63cc2980 error 4 in ld-2.28.so[7f958e2bf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:163422 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:163422 ppid:9691 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.18+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fee7a28870b
Code: Unable to access opcode bytes at RIP 0x7fee7a2886e1.
RSP: 002b:00007fff4cbc6cb8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fee7a28870b
RDX: 0000000000003832 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000005be5 R11: 0000000000000246 R12: 00007fff4cbc6ed8
R13: 0000000000000001 R14: 00007fff4cbc6d18 R15: 0000000000003832
13[211431]: segfault at 8 ip 00007f5bf26a9875 sp 00007fffc428c8e0 error 4 in ld-2.28.so[7f5bf2688000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 24757:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e5387e900 x1877479729174400/t0(0) o105->lustre-OST0003@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0003-osc-ffff923e0ec59800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1ed6:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x1786:0x0]/ may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x16e4:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x169d:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1e23:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1f36:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x17bd:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x17b3:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1f84:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x17ef:0x0]// may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1f22:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x17d6:0x0]/ may get corrupted (rc -108)
LustreError: 220716:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff923e0ec59800: namespace resource [0x380000400:0x24a:0x0].0x0 (ffff923ebc3b5500) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 220716:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
LustreError: 9833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790504901 with bad export cookie 1530551927601118604
LustreError: lustre-OST0003-osc-ffff923e0fed0800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x16e4:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x1742:0x0]// may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x16d7:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x1f29:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1839:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x16da:0x0]/ may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x171c:0x0]/ may get corrupted (rc -108)
LustreError: 20211:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923ea1ebc280 x1877479733368192/t0(0) o104->lustre-OST0002@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0002-osc-ffff923e0ec59800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2033:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x186f:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x184d:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x17f4:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x174b:0x0]/ may get corrupted (rc -108)
13[225848]: segfault at 8 ip 00007f4def75b875 sp 00007ffe4932cd70 error 4 in ld-2.28.so[7f4def73a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[226837]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[226837]: Failed to get EXE, ignoring: No such process
10[226645]: segfault at 8 ip 00007f696d7f5875 sp 00007ffce891bd70 error 4 in ld-2.28.so[7f696d7d4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 7820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790504936 with bad export cookie 1530551927604181468
LustreError: lustre-OST0001-osc-ffff923e0fed0800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1959:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x17cc:0x0]/ may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x200f:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x18fd:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x1833:0x0]// may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x20a9:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x1f0d:0x0]// may get corrupted (rc -108)
LustreError: 232989:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff923e0fed0800: namespace resource [0x300000401:0x27e:0x0].0x0 (ffff923d85efec00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 232989:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
LustreError: 235926:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0ec59800: inode [0x200000404:0x21c0:0x0] mdc close failed: rc = -13
LustreError: 235926:0:(file.c:252:ll_close_inode_openhandle()) Skipped 1 previous similar message
15[237711]: segfault at 8 ip 00007f6403fe5875 sp 00007fff993dde70 error 4 in ld-2.28.so[7f6403fc4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[238570]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[238570]: Failed to get EXE, ignoring: No such process
16[240023]: segfault at 8 ip 00007fdf26177875 sp 00007ffe01c74ed0 error 4 in ld-2.28.so[7fdf26156000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:209858 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:209858 ppid:9542 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
vvp_io_setattr_start+0x573/0x5f0 [lustre]
? init_wait_var_entry+0x50/0x50
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
ll_fallocate+0x884/0x9f0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7bb437870b
Code: Unable to access opcode bytes at RIP 0x7f7bb43786e1.
RSP: 002b:00007ffd567ff668 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000003 RCX: 00007f7bb437870b
RDX: 00000000000006f3 RSI: 0000000000000003 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000509e R11: 0000000000000246 R12: 00007ffd567ff888
R13: 0000000000000003 R14: 00007ffd567ff6c8 R15: 00000000000006f3
LustreError: 252688:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x240000404:0x1a77:0x0] mdc close failed: rc = -13
LustreError: 252688:0:(file.c:252:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000007dd5781e/0x153d9cde7348553a lrc: 3/0,0 mode: PW/PW res: [0x2c0000402:0x304:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 24576->49151) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0x153d9cde73485510 expref: 23288 pid: 20211 timeout: 1053 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 5 previous similar messages
LustreError: 146506:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e43492a00 x1877479760287360/t0(0) o104->lustre-OST0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0000-osc-ffff923e0ec59800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff923e0ec59800: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 3 previous similar messages
Lustre: Skipped 4 previous similar messages
LustreError: lustre-OST0000-osc-ffff923e0ec59800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x1cb1:0x0]/ may get corrupted (rc -5)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2383:0x0]/ may get corrupted (rc -5)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x1c24:0x0]// may get corrupted (rc -5)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x1b7e:0x0]/ may get corrupted (rc -5)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x233f:0x0]// may get corrupted (rc -5)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1be4:0x0]// may get corrupted (rc -5)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x1c8e:0x0]/ may get corrupted (rc -5)
LustreError: 257639:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff923e0ec59800: namespace resource [0x2c0000402:0x304:0x0].0x0 (ffff923ebc194600) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0000-osc-ffff923e0ec59800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 5 previous similar messages
LustreError: lustre-OST0001-osc-ffff923e0fed0800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x1c61:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2511:0x0]// may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x1c52:0x0]// may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x1d11:0x0]/ may get corrupted (rc -108)
14[269398]: segfault at 8 ip 00007f8cd9adf875 sp 00007ffc811b8f10 error 4 in ld-2.28.so[7f8cd9abe000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[270250]: segfault at 8 ip 00007f43a386b875 sp 00007ffc4dbef020 error 4 in ld-2.28.so[7f43a384a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[272481]: segfault at 55730bdbc000 ip 000055730bdbc000 sp 00007ffeb8bfff98 error 14 in 2[55730bfbc000+1000]
Code: Unable to access opcode bytes at RIP 0x55730bdbbfd6.
LustreError: 55645:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e1e8e9500 x1877479782355840/t0(0) o106->lustre-OST0001@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: lustre-OST0001-osc-ffff923e0ec59800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2844:0x0]// may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x1d96:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x28c4:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x1fbc:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x1f0a:0x0]// may get corrupted (rc -108)
LustreError: 284614:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff923e0ec59800: namespace resource [0x300000402:0x359:0x0].0x0 (ffff923e3c5b2500) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 284614:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 4 previous similar messages
16[287287]: segfault at 8 ip 00007f422d998875 sp 00007fff3634b010 error 4 in ld-2.28.so[7f422d977000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 14287:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 14287:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 261 previous similar messages
Lustre: 14287:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14287:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 261 previous similar messages
Lustre: 14287:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 14287:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 261 previous similar messages
Lustre: 14287:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 14287:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 261 previous similar messages
Lustre: 14287:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14287:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 261 previous similar messages
Lustre: 14287:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14287:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 261 previous similar messages
Lustre: 'ost_io' is processing requests too slowly, client may timeout. Late by 15s, missed 0 early replies (reqs waiting=0 active=5, at_estimate=304, delay=7ms)
LustreError: 302245:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0fed0800: inode [0x280000403:0x218b:0x0] mdc close failed: rc = -13
LustreError: 302245:0:(file.c:252:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Lustre: 6760:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
11[303941]: segfault at 8 ip 00007fd01cdb3875 sp 00007ffc8a83e560 error 4 in ld-2.28.so[7fd01cd92000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[305604]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[305604]: Failed to get EXE, ignoring: No such process
18[305933]: segfault at 8 ip 00007fc961a3b875 sp 00007ffe506b9250 error 4 in ld-2.28.so[7fc961a1a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[307106]: segfault at 8 ip 00007f5250ba4875 sp 00007ffdb28b81e0 error 4 in ld-2.28.so[7f5250b83000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: lustre-OST0002-osc-ffff923e0ec59800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2cbe:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2111:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x220d:0x0]/ may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2c4b:0x0]/ may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2b74:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x221e:0x0]/ may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x222b:0x0]// may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2246:0x0]/ may get corrupted (rc -108)
18[313345]: segfault at 8 ip 00007f88463a1875 sp 00007fff7abca060 error 4 in ld-2.28.so[7f8846380000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 1[319712] trap invalid opcode ip:55cb7ffd5956 sp:7ffd289790b8 error:0 in 1[55cb7ffd0000+7000]
16[320885]: segfault at 8 ip 00007f9d21aee875 sp 00007ffc11265bb0 error 4 in ld-2.28.so[7f9d21acd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 20030:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 513 < left 520, rollback = 2
Lustre: 20030:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 7 previous similar messages
LustreError: 6735:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e60ec8000 x1877479814379904/t0(0) o105->lustre-OST0001@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 6735:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: lustre-OST0001-osc-ffff923e0ec59800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2f48:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2ef1:0x0]/ may get corrupted (rc -5)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2382:0x0]/ may get corrupted (rc -5)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2ee3:0x0]// may get corrupted (rc -5)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x2f3c:0x0]// may get corrupted (rc -5)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2de7:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x23af:0x0]/ may get corrupted (rc -5)
LustreError: 323866:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff923e0ec59800: namespace resource [0x300000402:0x445:0x0].0x0 (ffff923e31eafd00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 323866:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 2 previous similar messages
LustreError: lustre-OST0002-osc-ffff923e0fed0800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x23fc:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2ee8:0x0]// may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x24b4:0x0]// may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2f2d:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2462:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x23ea:0x0]// may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x23cc:0x0]/ may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2e92:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x310e:0x0]// may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x2f54:0x0]/ may get corrupted (rc -108)
traps: 12[334576] trap invalid opcode ip:5564af77eea2 sp:7ffd7679ee58 error:0 in 12[5564af779000+7000]
systemd-coredump[335639]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[335639]: Failed to get EXE, ignoring: No such process
Lustre: lustre-OST0002: haven't heard from client 16a55e9a-6a6c-43a2-9a00-f43c107c6c8c (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp ffff923e5d149000, cur 1790505350 deadline 1790505349 last 1790505319
LustreError: lustre-OST0000-osc-ffff923e0fed0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x256f:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2545:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x25a7:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x314c:0x0]/ may get corrupted (rc -108)
LustreError: lustre-OST0002-osc-ffff923e0fed0800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
10[347276]: segfault at 8 ip 00007fe949ce1875 sp 00007ffcdaebf0b0 error 4 in ld-2.28.so[7fe949cc0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[353637]: segfault at 8 ip 00007fb269159875 sp 00007ffea9755840 error 4 in ld-2.28.so[7fb269138000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[354479]: segfault at 8 ip 00007f3911fd7875 sp 00007ffc94c758c0 error 4 in ld-2.28.so[7f3911fb6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: lustre-OST0001-osc-ffff923e0fed0800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x338d:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x287e:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x27de:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x28b2:0x0]// may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x28a9:0x0]/ may get corrupted (rc -108)
Lustre: 3357:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x33be:0x0]/ may get corrupted (rc -108)
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0003-osc-ffff923e0ec59800: granted 8437760 but already consumed 8785920
LustreError: lustre-OST0003-osc-ffff923e0ec59800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2714:0x0]/ may get corrupted (rc -108)
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x33be:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x284d:0x0]/ may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x27f2:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x34f4:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3516:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x288b:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x28c5:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x32ca:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2869:0x0]/ may get corrupted (rc -108)
Lustre: 3361:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2767:0x0]// may get corrupted (rc -108)
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0003-osc-ffff923e0fed0800: granted 8437760 but already consumed 10018816
LustreError: lustre-OST0003-osc-ffff923e0fed0800: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3362:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x29aa:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2869:0x0]// may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x29a3:0x0]/ may get corrupted (rc -108)
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x29da:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2869:0x0]// may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x29e6:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x29e2:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2a55:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x299a:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3738:0x0]// may get corrupted (rc -108)
Lustre: 3364:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2a3f:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x363e:0x0]/ may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x36bd:0x0]/ may get corrupted (rc -108)
Lustre: 3354:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2943:0x0]// may get corrupted (rc -108)
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0001-osc-ffff923e0ec59800: granted 8437760 but already consumed 9846784
LustreError: lustre-OST0001-osc-ffff923e0ec59800: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2a43:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3813:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2a1f:0x0]/ may get corrupted (rc -108)
Lustre: 3360:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x3641:0x0]// may get corrupted (rc -108)
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0002-osc-ffff923e0ec59800: granted 8437760 but already consumed 8884224
LustreError: lustre-OST0002-osc-ffff923e0ec59800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2a40:0x0]// may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x374b:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2af4:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x2a4d:0x0]/ may get corrupted (rc -108)
Lustre: 3363:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x369c:0x0]// may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000404:0x29e7:0x0]/ may get corrupted (rc -108)
LustreError: lustre-OST0000-osc-ffff923e0fed0800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000403:0x2b9c:0x0]/ may get corrupted (rc -108)
Lustre: 3358:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2a84:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2aba:0x0]/ may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x37ca:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x39d0:0x0]/ may get corrupted (rc -108)
Lustre: 3359:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x374d:0x0]// may get corrupted (rc -108)
1[398066]: segfault at 8 ip 00007fd0a2c0b875 sp 00007ffe6548df70 error 4 in ld-2.28.so[7fd0a2bea000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 8400:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff923e1552cd00 x1877479883186048/t0(0) o105->lustre-OST0000@0@lo:15/16 lens 392/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 8400:0:(client.c:1394:ptlrpc_import_delay_req()) Skipped 1 previous similar message
LustreError: 3351:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0000-osc-ffff923e0ec59800: granted 8437760 but already consumed 13131776
LustreError: lustre-OST0000-osc-ffff923e0ec59800: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3365:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2c5a:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x3a3b:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3c14:0x0]/ may get corrupted (rc -108)
Lustre: 3366:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3c80:0x0]// may get corrupted (rc -108)
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3a2b:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3c6e:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2d8c:0x0]/ may get corrupted (rc -108)
Lustre: 'ost_io' is processing requests too slowly, client may timeout. Late by 11s, missed 0 early replies (reqs waiting=0 active=7, at_estimate=304, delay=1ms)
Lustre: lustre-OST0000: haven't heard from client fa7905ca-1a9d-489b-b0a3-bd3f01ccf1c5 (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp ffff923e36fda000, cur 1790505558 deadline 1790505556 last 1790505526
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000a503bc2d/0x153d9cde73c15090 lrc: 3/0,0 mode: PW/PW res: [0x340000402:0x4f7:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 20480->53247) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x153d9cde73c14eec expref: 37 pid: 10600 timeout: 1584 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 6737:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 13 previous similar messages
LustreError: lustre-OST0002-osc-ffff923e0fed0800: operation ost_punch to node 0@lo failed: rc = -107
LustreError: Skipped 18 previous similar messages
Lustre: lustre-OST0002-osc-ffff923e0fed0800: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 14 previous similar messages
LustreError: lustre-OST0002-osc-ffff923e0fed0800: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3353:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2ea8:0x0]/ may get corrupted (rc -108)
Lustre: 3352:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x3c96:0x0]/ may get corrupted (rc -108)
Lustre: 3367:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2eb7:0x0]// may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000403:0x3cbd:0x0]/ may get corrupted (rc -108)
Lustre: 3355:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x200000404:0x3dc5:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x280000404:0x2f19:0x0]/ may get corrupted (rc -108)
Lustre: 3356:0:(llite_lib.c:4343:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.52@tcp:/lustre/fid: [0x240000403:0x2f1c:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff923e0fed0800: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 13 previous similar messages
Lustre: lustre-OST0000-osc-ffff923e0ec59800: disconnect after 55s idle
6[427508]: segfault at 8 ip 00007feb36c5f875 sp 00007ffca1da7c80 error 4 in ld-2.28.so[7feb36c3e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[428826]: segfault at 8 ip 00007fe8b95df875 sp 00007ffd965a0150 error 4 in ld-2.28.so[7fe8b95be000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 429661:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff923e0ec59800: inode [0x200000404:0x3fde:0x0] mdc close failed: rc = -13
LustreError: 429661:0:(file.c:252:ll_close_inode_openhandle()) Skipped 3 previous similar messages
Lustre: 20030:0:(osd_internal.h:1465:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 514 < left 520, rollback = 2
Lustre: 20030:0:(osd_internal.h:1465:osd_trans_exec_op()) Skipped 3 previous similar messages