general protection fault, probably for non-canonical address 0x201020201010d32: 0000 [#1] SMP DEBUG_PAGEALLOC
CPU: 0 PID: 307039 Comm: lfs Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:do_raw_spin_lock+0xa/0x190
Code: 8c 07 00 00 e9 c8 07 00 00 48 83 05 27 0f b6 02 01 31 db eb af 66 66 2e 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 89 e5 53 <8b> 47 04 48 89 fb 48 83 05 58 10 b6 02 01 48 83 05 30 0f b6 02 01
RSP: 0018:ffffa64851c4fd08 EFLAGS: 00010006
RAX: 0000000000000206 RBX: 0000000000000206 RCX: 000000028514919b
RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0201020201010d32
RBP: ffffa64851c4fd10 R08: ffff96b3e475cc6f R09: 0000000000000000
R10: 000000016da0d9d5 R11: ffff96b3e475d000 R12: 0000000000000000
R13: 0201020201010d32 R14: ffff96b3f0930c80 R15: 0000000000000003
FS: 00007f4f98b87680(0000) GS:ffff96b572200000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000558c3cc31f44 CR3: 000000015e988000 CR4: 00000000000006f0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __die_body+0x27/0xa0
? die_addr+0x55/0x90
? do_general_protection+0x200/0x420
? general_protection+0x1e/0x30
? do_raw_spin_lock+0xa/0x190
_raw_spin_lock_irqsave+0x67/0x80
try_to_wake_up+0x66/0x730
wake_up_process+0x1d/0x30
ll_deauthorize_statahead+0xe4/0x570 [lustre]
ll_file_release+0x4d0/0xcc0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f4f987547c5

Lustre: 8062:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8062:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8062:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8062:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8062:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8062:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6774:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff96b374bee900 x1879771652059520/t4294967372(0) o101->12d241fc-4950-4f76-93dd-e30636dd8d76@0@lo:634/0 lens 376/816 e 0 to 0 dl 1792689999 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10131:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff96b3e9a97000 x1879771652248832/t4294967506(0) o101->273d6c1a-0bc5-466c-af2e-3bf8d3bdbf32@0@lo:635/0 lens 384/864 e 0 to 0 dl 1792690000 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 10301:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 10301:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10301:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10301:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10301:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10301:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10301:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10301:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10301:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10301:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10301:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10301:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8063:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8063:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 8063:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8063:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8063:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8063:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8063:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8063:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8063:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8063:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 8063:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8063:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 1 previous similar message
2[14213]: segfault at 564447277000 ip 0000564447277000 sp 00007ffc2485d258 error 14 in 2[564447477000+1000]
Code: Unable to access opcode bytes at RIP 0x564447276fd6.
14[14277]: segfault at 8 ip 00007f377af26875 sp 00007ffdd3080dd0 error 4 in ld-2.28.so[7f377af05000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[14504]: segfault at 8 ip 00007f1e7451f875 sp 00007fff96267280 error 4 in ld-2.28.so[7f1e744fe000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[15480]: segfault at 0 ip 0000555920c5fb47 sp 00007ffd33ae8400 error 6 in 12[555920c5b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 12030:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12030:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 12030:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12030:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12030:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12030:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12030:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 12030:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12030:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12030:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 12030:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12030:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages
traps: 1[15559] trap invalid opcode ip:5640c03db45a sp:7ffd89d17a38 error:0 in 1[5640c03d6000+7000]
13[16267]: segfault at 8 ip 00007fba3e2b1875 sp 00007ffd028813e0 error 4 in ld-2.28.so[7fba3e290000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 12[16080] trap invalid opcode ip:5581c52d48ea sp:7ffec6630120 error:0 in 12[5581c52ce000+7000]
Lustre: 15650:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff96b3bc3e3100 x1879771656849152/t4294970017(0) o101->12d241fc-4950-4f76-93dd-e30636dd8d76@0@lo:654/0 lens 376/840 e 0 to 0 dl 1792690019 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 8064:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8064:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 8064:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8064:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8064:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8064:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8064:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 8064:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8064:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8064:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 8064:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8064:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
1[19806]: segfault at 8 ip 00007f3bc6c68875 sp 00007fff3886ca10 error 4 in ld-2.28.so[7f3bc6c47000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[22209]: segfault at 8 ip 00007fc9eb6f3875 sp 00007ffef3e14d30 error 4 in ld-2.28.so[7fc9eb6d2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[24783]: segfault at 8 ip 00007fa5ee1be875 sp 00007ffc0c34d2c0 error 4 in ld-2.28.so[7fa5ee19d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 12030:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 12030:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 12030:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 12030:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12030:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 12030:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12030:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 12030:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12030:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 12030:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 12030:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 12030:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: ll_ost_io00_006: service thread pid 14427 was inactive for 40.760 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_006 state:D stack:0 pid:14427 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
15[29579]: segfault at 8 ip 00007fd61f5df875 sp 00007fffe88be440 error 4 in ld-2.28.so[7fd61f5be000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[30683]: segfault at 8 ip 00007f92b163d875 sp 00007ffde34e86a0 error 4 in ld-2.28.so[7f92b161c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[31256]: Failed to get EXE, ignoring: No such process
Lustre: 10356:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10356:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 10356:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10356:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10356:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10356:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10356:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10356:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10356:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10356:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 10356:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10356:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 7 previous similar messages
16[34010]: segfault at 8 ip 00007fc144f72875 sp 00007ffc88ba21b0 error 4 in ld-2.28.so[7fc144f51000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[42526]: segfault at 8 ip 00007fe54e3ac875 sp 00007ffe3d277e40 error 4 in ld-2.28.so[7fe54e38b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[42921]: segfault at 8 ip 00007f8c3aeb7875 sp 00007fff3b3ae8f0 error 4 in ld-2.28.so[7f8c3ae96000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 39341:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 39341:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 39341:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 39341:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 39341:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 39341:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 39341:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 39341:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 39341:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 39341:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 39341:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 39341:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 5 previous similar messages
18[45400]: segfault at 0 ip 000055d0324ccb47 sp 00007ffe8adcb610 error 6 in 18[55d0324c8000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 000000002103f862/0x3caef347b7bb46a1 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0xd:0x0].0x0 rrc: 6 type: EXT [0->18446744073709551615] (req 24576->28671) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3caef347b7bb45d6 expref: 32 pid: 10680 timeout: 191 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff96b34c991000: operation ost_read to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff96b34c991000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff96b34c991000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3149:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x34c:0x0]// may get corrupted (rc -108)
Lustre: 3149:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x3e4:0x0]// may get corrupted (rc -108)
Lustre: 3143:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x2fe:0x0]/ may get corrupted (rc -108)
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x39b:0x0]// may get corrupted (rc -108)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x3d4:0x0]/ may get corrupted (rc -108)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x3c3:0x0]/ may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x3e4:0x0]/ may get corrupted (rc -108)
Lustre: 3147:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x311:0x0]/ may get corrupted (rc -108)
Lustre: 3147:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x3de:0x0]/ may get corrupted (rc -108)
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x395:0x0]/ may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x342:0x0]/ may get corrupted (rc -108)
LustreError: 48696:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff96b34c991000: namespace resource [0x300000400:0xd:0x0].0x0 (ffff96b350c7ec00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0002-osc-ffff96b34c991000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_006: service thread pid 14427 completed after 109.971s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
3[56478]: segfault at 1b30 ip 0000000000001b30 sp 00007ffeb3f03a60 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
13[59434]: segfault at 8 ip 00007f1290d71875 sp 00007fffff1b98c0 error 4 in ld-2.28.so[7f1290d50000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000050116fe2/0x3caef347b7c8318e lrc: 3/0,0 mode: PW/PW res: [0x340000402:0x3b:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8192->36863) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x3caef347b7c83179 expref: 1622 pid: 10742 timeout: 238 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 21039:0:(ldlm_lockd.c:2556:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1792690151 with bad export cookie 4372699777573855040
LustreError: 14980:0:(client.c:1411:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff96b3ae6ac600 x1879771693957760/t0(0) o106->lustre-OST0003@0@lo:15/16 lens 328/280 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: lustre-OST0003-osc-ffff96b3450d5000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0003-osc-ffff96b3450d5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
LustreError: lustre-OST0003-osc-ffff96b3450d5000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x486:0x0]/ may get corrupted (rc -108)
Lustre: 3147:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x53d:0x0]// may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x5f3:0x0]/ may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x64a:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x640:0x0]// may get corrupted (rc -108)
LustreError: 63365:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff96b3450d5000: namespace resource [0x340000400:0x6a:0x0].0x0 (ffff96b347f74d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 63365:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff96b3450d5000: Connection restored to 0@lo (at 0@lo)
Lustre: 49129:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 49129:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 49129:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 49129:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 49129:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 49129:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 49129:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 49129:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 49129:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 49129:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 49129:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 49129:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 17 previous similar messages
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000c3552ef7/0x3caef347b7c9db45 lrc: 3/0,0 mode: PW/PW res: [0x380000401:0x3d:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 28672->61439) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x3caef347b7c9da81 expref: 437 pid: 10722 timeout: 249 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff96b3450d5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff96b3450d5000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff96b3450d5000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x561:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff96b3450d5000: Connection restored to 0@lo (at 0@lo)
Lustre: 6786:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 520, rollback = 2
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000f177bcab/0x3caef347b7d4eae8 lrc: 3/0,0 mode: PW/PW res: [0x300000402:0x6d:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8192->20479) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x3caef347b7d4ea2b expref: 315 pid: 10722 timeout: 289 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff96b3450d5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff96b3450d5000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff96b3450d5000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x5d7:0x0]// may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x85f:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x6a8:0x0]// may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x6d2:0x0]/ may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x604:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x8c5:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x685:0x0]// may get corrupted (rc -108)
LustreError: 79066:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff96b3450d5000: namespace resource [0x300000401:0x5b:0x0].0x0 (ffff96b3e0e0dd00) refcount nonzero (2) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0002-osc-ffff96b3450d5000: Connection restored to 0@lo (at 0@lo)
LustreError: lustre-OST0002-osc-ffff96b34c991000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x850:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x65f:0x0]/ may get corrupted (rc -108)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x704:0x0]/ may get corrupted (rc -108)
Lustre: 3149:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x861:0x0]/ may get corrupted (rc -108)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x8b6:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x66d:0x0]/ may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x78a:0x0]/ may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x617:0x0]/ may get corrupted (rc -108)
0[81255]: segfault at 8 ip 00007f0df1082875 sp 00007ffd8b855c80 error 4 in ld-2.28.so[7f0df1061000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0003_UUID lock: 0000000058d4fe91/0x3caef347b7dab5f3 lrc: 3/0,0 mode: PW/PW res: [0x340000400:0x6a:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 24576->40959) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3caef347b7dab5ec expref: 569 pid: 10722 timeout: 312 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 1 previous similar message
LustreError: lustre-OST0003-osc-ffff96b34c991000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0003-osc-ffff96b34c991000: Connection to lustre-OST0003 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0003-osc-ffff96b34c991000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x7e9:0x0]/ may get corrupted (rc -108)
Lustre: 3147:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x719:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x815:0x0]// may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x74b:0x0]// may get corrupted (rc -108)
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x700:0x0]// may get corrupted (rc -108)
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x6f0:0x0]// may get corrupted (rc -108)
Lustre: 3149:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x836:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x6ec:0x0]/ may get corrupted (rc -108)
LustreError: 85104:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0003-osc-ffff96b34c991000: namespace resource [0x340000400:0x6a:0x0].0x0 (ffff96b381de2d00) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0003-osc-ffff96b34c991000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 1 previous similar message
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000004c254b38/0x3caef347b7df970a lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x7f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 16384->20479) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x3caef347b7df96ee expref: 895 pid: 14980 timeout: 336 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0000-osc-ffff96b34c991000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0000-osc-ffff96b34c991000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0000-osc-ffff96b34c991000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x74a:0x0]/ may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x808:0x0]// may get corrupted (rc -108)
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x8f3:0x0]// may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x804:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x84c:0x0]/ may get corrupted (rc -108)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x93a:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x8d8:0x0]// may get corrupted (rc -108)
Lustre: 3149:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x946:0x0]/ may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xa52:0x0]// may get corrupted (rc -108)
LustreError: 92562:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0000-osc-ffff96b34c991000: namespace resource [0x2c0000401:0x79:0x0].0x0 (ffff96b36fdf4700) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0000-osc-ffff96b34c991000: Connection restored to 0@lo (at 0@lo)
LustreError: lustre-OST0000-osc-ffff96b3450d5000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x88c:0x0]/ may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x856:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x7cb:0x0]/ may get corrupted (rc -108)
Lustre: 3147:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xaa8:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x79a:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x7be:0x0]// may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xaee:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x74a:0x0]/ may get corrupted (rc -108)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x7c2:0x0]// may get corrupted (rc -108)
LustreError: 3142:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0003-osc-ffff96b3450d5000: granted 8437760 but already consumed 11137024
LustreError: lustre-OST0003-osc-ffff96b3450d5000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x805:0x0]/ may get corrupted (rc -5)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xac4:0x0]/ may get corrupted (rc -5)
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xb19:0x0]/ may get corrupted (rc -5)
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x8dd:0x0]// may get corrupted (rc -5)
Lustre: 3149:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x7f3:0x0]/ may get corrupted (rc -5)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x8e7:0x0]/ may get corrupted (rc -5)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x865:0x0]/ may get corrupted (rc -5)
Lustre: 3157:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xafb:0x0]// may get corrupted (rc -108)
Lustre: 6785:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 6785:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000001b7cb1f6/0x3caef347b7e6a8d0 lrc: 3/0,0 mode: PW/PW res: [0x380000400:0xa2:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 4096->8191) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x3caef347b7e6a89f expref: 31 pid: 10742 timeout: 364 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) Skipped 2 previous similar messages
LustreError: lustre-OST0001-osc-ffff96b3450d5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 2 previous similar messages
Lustre: lustre-OST0001-osc-ffff96b3450d5000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: lustre-OST0001-osc-ffff96b3450d5000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0xa28:0x0]/ may get corrupted (rc -108)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x95e:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xacc:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xbed:0x0]// may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x9c5:0x0]// may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x80c:0x0]/ may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x8fd:0x0]// may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0xb77:0x0]/ may get corrupted (rc -108)
LustreError: 101135:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff96b3450d5000: namespace resource [0x380000401:0xea:0x0].0x0 (ffff96b392a25500) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 101135:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 3 previous similar messages
Lustre: lustre-OST0001-osc-ffff96b3450d5000: Connection restored to 0@lo (at 0@lo)
Lustre: Skipped 2 previous similar messages
11[102892]: segfault at 8 ip 00007fe067d96875 sp 00007ffeef3014a0 error 4 in ld-2.28.so[7fe067d75000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 8063:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8063:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 71 previous similar messages
Lustre: 8063:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8063:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 8063:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8063:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 8063:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8063:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 8063:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8063:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 75 previous similar messages
Lustre: 8063:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8063:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 75 previous similar messages
19[106741]: segfault at 8 ip 00007f1f336ae875 sp 00007ffdaf65cac0 error 4 in ld-2.28.so[7f1f3368d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000000d1cf9a1/0x3caef347b7eea754 lrc: 3/0,0 mode: PW/PW res: [0x380000402:0xcd:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 8192->32767) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3caef347b7eea731 expref: 3033 pid: 8059 timeout: 397 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff96b34c991000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-ffff96b34c991000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff96b34c991000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xd4a:0x0]// may get corrupted (rc -5)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xd61:0x0]// may get corrupted (rc -5)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0xb8a:0x0]// may get corrupted (rc -108)
Lustre: 3155:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xc44:0x0]/ may get corrupted (rc -108)
Lustre: 3154:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xc08:0x0]/ may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0xd8d:0x0]// may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0xa0b:0x0]// may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x97b:0x0]// may get corrupted (rc -108)
LustreError: 112638:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff96b34c991000: namespace resource [0x380000401:0x105:0x0].0x0 (ffff96b34c8d6b00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 112638:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 3 previous similar messages
Lustre: lustre-OST0001-osc-ffff96b34c991000: Connection restored to 0@lo (at 0@lo)
19[120733]: segfault at 8 ip 00007f13b997f875 sp 00007fff6f8a6d00 error 4 in ld-2.28.so[7f13b995e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 13[127896] trap invalid opcode ip:55921056e8dc sp:7ffe0fc9ae28 error:0 in 7[559210568000+7000]
traps: 7[127545] trap invalid opcode ip:56338bb838dc sp:7fffac9bd958 error:0 in 7[56338bb7d000+7000]
5[133171]: segfault at 8 ip 00007fe127437875 sp 00007ffdeba7d5a0 error 4 in ld-2.28.so[7fe127416000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 133171:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff96b3450d5000: inode [0x200000403:0x1070:0x0] mdc close failed: rc = -13
6[142672]: segfault at 8 ip 00007f3cff521875 sp 00007ffd4ea65ff0 error 4 in ld-2.28.so[7f3cff500000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task fallocate:95061 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:95061 ppid:9418 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fabed45970b
Code: Unable to access opcode bytes at RIP 0x7fabed4596e1.
RSP: 002b:00007fffbe280fb8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fabed45970b
RDX: 0000000000005d66 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000039f3 R11: 0000000000000246 R12: 00007fffbe2811d8
R13: 0000000000000001 R14: 00007fffbe281018 R15: 0000000000005d66
19[146546]: segfault at 0 ip 000055b72ddb7b47 sp 00007fff763fe080 error 6 in 19[55b72ddb3000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3[148894]: segfault at 8 ip 00007f1fa58d3875 sp 00007ffc78a91df0 error 4 in ld-2.28.so[7f1fa58b2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 0000000011fe0747/0x3caef347b80e5b3a lrc: 3/0,0 mode: PW/PW res: [0x300000401:0x11f:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 24576->36863) gid 0 flags: 0x60000400010020 nid: 0@lo remote: 0x3caef347b80e5b33 expref: 3707 pid: 14886 timeout: 526 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff96b34c991000: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff96b34c991000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: Skipped 1 previous similar message
LustreError: lustre-OST0002-osc-ffff96b34c991000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0xe24:0x0]// may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x132b:0x0]/ may get corrupted (rc -108)
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0xe7f:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0xf62:0x0]/ may get corrupted (rc -108)
Lustre: 3143:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0xe4f:0x0]/ may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0xe7a:0x0]/ may get corrupted (rc -108)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0xe6d:0x0]/ may get corrupted (rc -108)
LustreError: 156343:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff96b34c991000: namespace resource [0x300000401:0x17e:0x0].0x0 (ffff96b381550400) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 156343:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-ffff96b34c991000: Connection restored to 0@lo (at 0@lo)
LustreError: 157385:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff96b34c991000: inode [0x240000404:0xe7a:0x0] mdc close failed: rc = -13
LustreError: 158563:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff96b34c991000: inode [0x240000403:0xedc:0x0] mdc close failed: rc = -13
LustreError: 159347:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff96b34c991000: inode [0x240000403:0xed9:0x0] mdc close failed: rc = -13
Lustre: 6785:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 516 < left 520, rollback = 2
Lustre: 6785:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
INFO: task fallocate:95061 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:95061 ppid:9418 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fabed45970b
Code: Unable to access opcode bytes at RIP 0x7fabed4596e1.
RSP: 002b:00007fffbe280fb8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fabed45970b
RDX: 0000000000005d66 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000039f3 R11: 0000000000000246 R12: 00007fffbe2811d8
R13: 0000000000000001 R14: 00007fffbe281018 R15: 0000000000005d66
INFO: task fallocate:104697 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:104697 ppid:9595 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fc7638c370b
Code: Unable to access opcode bytes at RIP 0x7fc7638c36e1.
RSP: 002b:00007ffe9af97d38 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fc7638c370b
RDX: 0000000000007729 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000000d9 R11: 0000000000000246 R12: 00007ffe9af97f58
R13: 0000000000000000 R14: 00007ffe9af97d98 R15: 0000000000007729
INFO: task fallocate:109728 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:109728 ppid:9655 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f475310570b
Code: Unable to access opcode bytes at RIP 0x7f47531056e1.
RSP: 002b:00007ffde875a388 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f475310570b
RDX: 0000000000003cfa RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 00000000000001a1 R11: 0000000000000246 R12: 00007ffde875a5a8
R13: 0000000000000001 R14: 00007ffde875a3e8 R15: 0000000000003cfa
INFO: task chown:109857 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:109857 ppid:9621 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fe28990cfbe
Code: Unable to access opcode bytes at RIP 0x7fe28990cf94.
RSP: 002b:00007ffd53733368 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffd537335a0 RCX: 00007fe28990cfbe
RDX: 0000000000007b99 RSI: 00005572a1b9dd20 RDI: 00000000ffffff9c
RBP: 00005572a1b9ef50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 000000000000293f R11: 0000000000000246 R12: 00000000ffffffff
R13: 00005572a1b9efc8 R14: 0000000000007b99 R15: 000000000000293f
INFO: task chmod:110316 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:110316 ppid:9622 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fe8f8518258
Code: Unable to access opcode bytes at RIP 0x7fe8f851822e.
RSP: 002b:00007fff60b62218 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fe8f8518258
RDX: 0000000000000e94 RSI: 0000559bcdc92500 RDI: 00000000ffffff9c
RBP: 0000559bcdc92470 R08: 0000000000000000 R09: 0000000000000e94
R10: 0000000000000fff R11: 0000000000000246 R12: 0000559bcc75cbb4
R13: 0000559bcdc92500 R14: 0000559bcdc93730 R15: 0000000000008000
INFO: task chown:110389 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:110389 ppid:9563 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f2b9ffe3fbe
Code: Unable to access opcode bytes at RIP 0x7f2b9ffe3f94.
RSP: 002b:00007ffe4feea2b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffe4feea4f0 RCX: 00007f2b9ffe3fbe
RDX: 0000000000007bad RSI: 000055fe4df9bd20 RDI: 00000000ffffff9c
RBP: 000055fe4df9cf50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000003fe3 R11: 0000000000000246 R12: 00000000ffffffff
R13: 000055fe4df9cfc8 R14: 0000000000007bad R15: 0000000000003fe3
INFO: task fallocate:115623 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:115623 ppid:9806 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f73dcd8c70b
Code: Unable to access opcode bytes at RIP 0x7f73dcd8c6e1.
RSP: 002b:00007ffdb5b477f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f73dcd8c70b
RDX: 0000000000006e43 RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 000000000000729b R11: 0000000000000246 R12: 00007ffdb5b47a18
R13: 0000000000000000 R14: 00007ffdb5b47858 R15: 0000000000006e43
INFO: task fallocate:122586 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:122586 ppid:9838 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7468af770b
Code: Unable to access opcode bytes at RIP 0x7f7468af76e1.
RSP: 002b:00007ffe40470f68 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f7468af770b
RDX: 00000000000044aa RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004925 R11: 0000000000000246 R12: 00007ffe40471188
R13: 0000000000000001 R14: 00007ffe40470fc8 R15: 00000000000044aa
INFO: task fallocate:127592 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:127592 ppid:9816 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fb2e79e770b
Code: Unable to access opcode bytes at RIP 0x7fb2e79e76e1.
RSP: 002b:00007ffdb8cde048 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fb2e79e770b
RDX: 0000000000003b54 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001b61 R11: 0000000000000246 R12: 00007ffdb8cde268
R13: 0000000000000001 R14: 00007ffdb8cde0a8 R15: 0000000000003b54
Lustre: 165019:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 165019:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 221 previous similar messages
Lustre: 165019:0:(osd_handler.c:2097:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 165019:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 223 previous similar messages
Lustre: 165019:0:(osd_handler.c:2104:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 165019:0:(osd_handler.c:2104:osd_trans_dump_creds()) Skipped 223 previous similar messages
Lustre: 165019:0:(osd_handler.c:2114:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 165019:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 223 previous similar messages
Lustre: 165019:0:(osd_handler.c:2121:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 165019:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 223 previous similar messages
Lustre: 165019:0:(osd_handler.c:2128:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 165019:0:(osd_handler.c:2128:osd_trans_dump_creds()) Skipped 223 previous similar messages
Lustre: ll_ost_io00_016: service thread pid 49339 was inactive for 381.465 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_016 state:D stack:0 pid:49339 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 216156:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff96b34c991000: inode [0x200000404:0x1bbb:0x0] mdc close failed: rc = -13
Lustre: ll_ost_io00_004: service thread pid 10356 was inactive for 384.920 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_004 state:D stack:0 pid:10356 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
? slab_post_alloc_hook+0x6d/0x390
? kmem_cache_alloc_trace+0xd0/0x440
down_write+0x84/0xd0
osd_trunc_lock+0x1f8/0x330 [osd_ldiskfs]
osd_declare_fallocate+0x4ab/0x740 [osd_ldiskfs]
dt_declare_fallocate+0x30/0xa0 [obdclass]
ofd_object_fallocate+0x485/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_io00_001: service thread pid 8063 was inactive for 401.060 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: ll_ost_io00_003: service thread pid 10301 was inactive for 382.603 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
task:ll_ost_io00_001 state:R running task stack:0 pid:8063 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_io00_015: service thread pid 49227 was inactive for 386.445 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
8[237916]: segfault at 8 ip 00007f1b40834875 sp 00007ffc030984d0 error 4 in ld-2.28.so[7f1b40813000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: ll_ost_io00_010: service thread pid 34946 was inactive for 401.075 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
Lustre: ll_ost_io00_014: service thread pid 49129 was inactive for 387.625 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: 6787:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 6787:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
7[267886]: segfault at 8 ip 00007f958b6f2875 sp 00007ffece672ae0 error 4 in ld-2.28.so[7f958b6d1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 224970:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (4/4), not sending early reply req@ffff96b3d9bd9880 x1879771720447104/t0(0) o22->12d241fc-4950-4f76-93dd-e30636dd8d76@0@lo:737/0 lens 440/432 e 15 to 0 dl 1792690857 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: ll_ost_io00_011: service thread pid 35012 was inactive for 393.479 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 6759:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 364s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: 000000001aff77be/0x3caef347b83d1f5c lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0x1fb:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 20480->24575) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0x3caef347b83d1efa expref: 120 pid: 113683 timeout: 720 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff96b3450d5000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 1 previous similar message
Lustre: lustre-OST0001-osc-ffff96b3450d5000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff96b3450d5000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x1e21:0x0]// may get corrupted (rc -5)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1b6c:0x0]// may get corrupted (rc -5)
Lustre: 3154:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x1e31:0x0]// may get corrupted (rc -5)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x1e87:0x0]/ may get corrupted (rc -5)
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x25d6:0x0]/ may get corrupted (rc -5)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1b40:0x0]/ may get corrupted (rc -5)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x263c:0x0]// may get corrupted (rc -5)
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x26a0:0x0]/ may get corrupted (rc -5)
LustreError: lustre-OST0003-osc-ffff96b34c991000: This client was evicted by lustre-OST0003; in progress operations using this service will fail.
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1c17:0x0]/ may get corrupted (rc -5)
LustreError: 283162:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff96b3450d5000: namespace resource [0x380000402:0x354:0x0].0x0 (ffff96b39f53ba00) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 283162:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x1d82:0x0]// may get corrupted (rc -5)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x1a9d:0x0]/ may get corrupted (rc -108)
Lustre: 3152:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0xa22:0x0]/ may get corrupted (rc -108)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x1b84:0x0]// may get corrupted (rc -108)
Lustre: 3153:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x1e14:0x0]/ may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x24d3:0x0]// may get corrupted (rc -108)
Lustre: 3148:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x247e:0x0]/ may get corrupted (rc -108)
Lustre: 3156:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x26a3:0x0]/ may get corrupted (rc -108)
Lustre: 3150:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000404:0x1b65:0x0]// may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x246f:0x0]/ may get corrupted (rc -108)
Lustre: 3151:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1bf9:0x0]/ may get corrupted (rc -108)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x2460:0x0]/ may get corrupted (rc -108)
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000404:0x1ec7:0x0]/ may get corrupted (rc -108)
LustreError: lustre-OST0000-osc-ffff96b3450d5000: This client was evicted by lustre-OST0000; in progress operations using this service will fail.
Lustre: 205037:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (5/5), not sending early reply req@ffff96b3e452f700 x1879771733075840/t0(0) o22->12d241fc-4950-4f76-93dd-e30636dd8d76@0@lo:28/0 lens 440/432 e 8 to 0 dl 1792690903 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: 3158:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1b6d:0x0]/ may get corrupted (rc -108)
Lustre: 3145:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000403:0x25cd:0x0]/ may get corrupted (rc -108)
Lustre: 3147:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x200000404:0x24ee:0x0]/ may get corrupted (rc -108)
Lustre: 3146:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1bda:0x0]/ may get corrupted (rc -108)
Lustre: 3157:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x1dc8:0x0]/ may get corrupted (rc -108)
Lustre: 3144:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x280000403:0x1d70:0x0]/ may get corrupted (rc -108)
Lustre: 3143:0:(llite_lib.c:4334:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.112@tcp:/lustre/fid: [0x240000403:0x1ba2:0x0]// may get corrupted (rc -108)
Lustre: lustre-OST0000-osc-ffff96b3450d5000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_001: service thread pid 8063 completed after 595.562s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_005: service thread pid 12030 completed after 422.785s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_003: service thread pid 10301 completed after 577.898s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 10356:0:(service.c:2603:ptlrpc_server_handle_request()) @@@ Request took longer than estimated (238/375s); client may timeout req@ffff96b393b9cd00 x1879771728421632/t0(0) o22->12d241fc-4950-4f76-93dd-e30636dd8d76@0@lo:405/0 lens 440/432 e 0 to 0 dl 1792690525 ref 1 fl Complete:/600/0 rc -2/-2 job:'fallocate.0' uid:0 gid:0 projid:0
Lustre: ll_ost_io00_007: service thread pid 14902 completed after 588.477s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_017: service thread pid 49423 completed after 425.140s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_004: service thread pid 10356 completed after 612.865s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_011: service thread pid 35012 completed after 425.329s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_010: service thread pid 34946 completed after 531.618s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
16[284179]: segfault at 8 ip 00007f8834b8b875 sp 00007fffcef30f40 error 4 in ld-2.28.so[7f8834b6a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: ll_ost_io00_009: service thread pid 17431 completed after 562.582s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_io00_014: service thread pid 49129 completed after 487.807s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
2[288856]: segfault at 0 ip 0000558498ad4b47 sp 00007ffc36cabf00 error 6 in 2[558498ad0000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9[288327]: segfault at 8 ip 00007f4855f99875 sp 00007ffee5c46750 error 4 in ld-2.28.so[7f4855f78000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 14902:0:(service.c:1628:ptlrpc_at_send_early_reply()) @@@ Could not add any time (1/1), not sending early reply req@ffff96b3cbfdaa00 x1879771747040768/t0(0) o22->12d241fc-4950-4f76-93dd-e30636dd8d76@0@lo:76/0 lens 440/432 e 4 to 0 dl 1792690951 ref 2 fl Interpret:/600/0 rc 0/0 job:'fallocate.0' uid:0 gid:0 projid:0
8[304480]: segfault at 0 ip 000055b315b04b47 sp 00007ffdc5f2a1c0 error 6 in 8[55b315b00000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00