Editing crashreport #67850

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( list_empty(&ctx->sax_sai_list) ) failedll_sax_putll_statahead_thread
kthread
ret_from_fork
35

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashLink and date
racer test 1: racer on clients: centos-90.localnet DURATION=2700
LustreError: 2408120:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 2408120:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 13 PID: 2408120 Comm: ll_sa_2407423 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 1743042:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1743042:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1743042:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1743042:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1743042:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1743042:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
6[1745231]: segfault at 0 ip 0000555893cd3b47 sp 00007ffc0f709800 error 6 in 6[555893ccf000+7000]
Code: Unable to access opcode bytes at RIP 0x555893cd3b1d.
Lustre: 1743041:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1743041:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1743041:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1743041:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1743041:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1743041:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1743041:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1743041:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1743041:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1743041:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1743041:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1743041:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1745090:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1745090:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745090:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1745090:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745028:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff987611769880 x1875447761634432/t4294970299(0) o101->17af992b-2296-426c-a7f1-3b3a99302c25@0@lo:116/0 lens 376/816 e 0 to 0 dl 1788566426 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
2[1748459]: segfault at 8 ip 00007f2baadf0875 sp 00007fffdd58b810 error 4 in ld-2.28.so[7f2baadcf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1745393:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1745393:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745393:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1745393:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 1745090:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1745090:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745090:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1745090:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 5 previous similar messages
19[1754349]: segfault at 0 ip 00005585ac20ab47 sp 00007fff6c377180 error 6 in 19[5585ac206000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 1754349:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000401:0x361:0x0] mdc close failed: rc = -13
Lustre: 1743041:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1743041:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 1743041:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1743041:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1743041:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1743041:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1743041:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1743041:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1743041:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1743041:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1743041:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1743041:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 9 previous similar messages
2[1757880]: segfault at 8 ip 00007f49fedd4875 sp 00007ffef6937d00 error 4 in ld-2.28.so[7f49fedb3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[1758748]: segfault at 8 ip 00007fbcb0d1d875 sp 00007ffd269f4150 error 4 in ld-2.28.so[7fbcb0cfc000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1745090:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1745090:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745090:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1745090:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1745090:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745090:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1745394:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1745394:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745394:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745394:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1745394:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745394:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745394:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 1745077:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 1742783:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1742783:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 1780903:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875cf553000: inode [0x200000402:0xb20:0x0] mdc close failed: rc = -13
16[1782556]: segfault at 8 ip 00007f2b07262875 sp 00007ffdff215ca0 error 4 in ld-2.28.so[7f2b07241000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1743043:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1743043:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1743043:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1743043:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1743043:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1743043:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1743043:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 23 previous similar messages
0[1789830]: segfault at 8 ip 00007f2e00976875 sp 00007ffc7030f170 error 4 in ld-2.28.so[7f2e00955000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[1792299]: segfault at 7fff78703e3a ip 000055a749fed2cb sp 00007fff6061f968 error 6 in 12[55a749fe7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 41 0e <10> 86 02 44 0e 18 83 03 46 0e 20 02 54 0a 0e 18 41 0e 10 41 0e 08
4[1796188]: segfault at 8 ip 00007f941a9a2875 sp 00007ffe09c52a10 error 4 in ld-2.28.so[7f941a981000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[1807971]: segfault at 0 ip 00005577d5a6ab47 sp 00007ffd2ee1eb90 error 6 in 5[5577d5a66000+7000]
Code: Unable to access opcode bytes at RIP 0x5577d5a6ab1d.
LustreError: 1815254:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000402:0x1675:0x0] mdc close failed: rc = -13
Lustre: 1743042:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1743042:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 127 previous similar messages
Lustre: 1743042:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1743042:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 1743042:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1743042:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 1743042:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1743042:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 1743042:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1743042:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 127 previous similar messages
Lustre: 1743042:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1743042:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 127 previous similar messages
5[1831280]: segfault at 8 ip 00007f8aa56c6875 sp 00007fff23b3ce80 error 4 in ld-2.28.so[7f8aa56a5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[1831373]: segfault at 0 ip 000055ef9c52db47 sp 00007fffb820ad90 error 6 in 14[55ef9c529000+7000]
Code: Unable to access opcode bytes at RIP 0x55ef9c52db1d.
18[1851976]: segfault at 8 ip 00007f76d53e7875 sp 00007ffe685806e0 error 4 in ld-2.28.so[7f76d53c6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[1858869]: segfault at 0 ip 00005559d3bd235c sp 00007ffc69764078 error 6 in 9[5559d3bcd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4e 40 00 00 00 00 00 20 ac 40 00 00 00 00 00 18 f5 40 38 2e 33 30 <00> 43 00 77 72 69 74 65 20 65 72 72 6f 72 00 25 73 3a 20 25 73 00
17[1860506]: segfault at 8 ip 00007fd5b9011875 sp 00007ffddcee5d00 error 4 in ld-2.28.so[7fd5b8ff0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[1868067]: segfault at 8 ip 00007f93ba76f875 sp 00007ffce8923920 error 4 in ld-2.28.so[7f93ba74e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[1874863]: segfault at 8 ip 00007fb56b9e6875 sp 00007ffcb8cd9860 error 4 in ld-2.28.so[7fb56b9c5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[1878118]: segfault at 8 ip 00007f58c8a0b875 sp 00007fff67a23310 error 4 in ld-2.28.so[7f58c89ea000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1870134:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_1869201 LIST => FNAME no wakeup.
Lustre: 1745394:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1745394:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 233 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745394:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 233 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745394:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 233 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1745394:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 233 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745394:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 233 previous similar messages
Lustre: 1745394:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745394:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 233 previous similar messages
18[1886241]: segfault at 8 ip 00007f834e027875 sp 00007ffe61aad390 error 4 in ld-2.28.so[7f834e006000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1742783:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1742783:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
16[1893099]: segfault at 8 ip 00007fdeb70fd875 sp 00007ffc9a3ccae0 error 4 in ld-2.28.so[7fdeb70dc000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[1901260]: segfault at 0 ip 00005649cc47db47 sp 00007fff44daedb0 error 6 in 18[5649cc479000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5[1905066]: segfault at 0 ip 00005607f34bbb47 sp 00007ffdd8016e50 error 6 in 5[5607f34b7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4[1916146]: segfault at 8 ip 00007f4b74d2d875 sp 00007ffde8cc4ab0 error 4 in ld-2.28.so[7f4b74d0c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1923381:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: mdt00_017: service thread pid 1751297 was inactive for 40.501 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: mdt00_012: service thread pid 1745581 was inactive for 40.524 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_021: service thread pid 1751437 was inactive for 40.505 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt00_017 state:I
task:mdt00_022 state:I stack:0 pid:1760448 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
Lustre: Skipped 1 previous similar message
task:mdt00_021 state:I
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
stack:0 pid:1751297 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
stack:0 pid:1751437 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_intent_policy+0x151/0x680 [mdt]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
mdt_intent_policy+0x151/0x680 [mdt]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: mdt00_013: service thread pid 1745588 was inactive for 44.087 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_019: service thread pid 1751433 was inactive for 44.088 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 10 previous similar messages
Lustre: Skipped 15 previous similar messages
Lustre: ll_ost_out00_00: service thread pid 1743047 was inactive for 73.212 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
LustreError: 1742758:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff9876747a7000/0x347fd8e740d847ff lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x3b23:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x347fd8e740d846d9 expref: 1318 pid: 1774089 timeout: 9481 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_003: service thread pid 1743317 completed after 102.099s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 1743973 completed after 102.103s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_016: service thread pid 1745822 completed after 101.426s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_out00_00: service thread pid 1743047 completed after 81.412s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1745581:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000063ce9c22 ns: mdt-lustre-MDT0000_UUID lock: ffff98760e74b600/0x347fd8e740d84e57 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 25 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x347fd8e740d84e49 expref: 1310 pid: 1745581 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_000: service thread pid 1742768 completed after 101.435s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 1742770 completed after 101.423s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 1745023 completed after 101.998s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_004: service thread pid 1789960 completed after 102.147s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_001: service thread pid 1742769 completed after 101.970s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_015: service thread pid 1745673 completed after 101.456s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9875c514b800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff9875c514b800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_024: service thread pid 1907284 completed after 101.445s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_022: service thread pid 1760448 completed after 101.978s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 1744959 completed after 101.446s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 1751437 completed after 101.960s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9875c514b800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt00_011: service thread pid 1745068 completed after 102.111s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_013: service thread pid 1745588 completed after 101.450s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 1751297 completed after 101.962s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_out00_00: service thread pid 1774089 completed after 102.172s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_012: service thread pid 1745581 completed after 101.987s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1925364:0:(file.c:6219:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
Lustre: mdt00_019: service thread pid 1751433 completed after 101.463s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1925467:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000401:0x1:0x0] mdc close failed: rc = -108
Lustre: mdt00_006: service thread pid 1744989 completed after 102.133s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_018: service thread pid 1751430 completed after 102.118s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1745068:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff987619ec0700 x1875447882880896/t0(0) o104->lustre-MDT0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: lustre-MDT0000-mdc-ffff9875c514b800: Connection restored to 0@lo (at 0@lo)
6[1932580]: segfault at 8 ip 00007fb4ca315875 sp 00007ffd436c2110 error 4 in ld-2.28.so[7fb4ca2f4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 3[1934565] trap invalid opcode ip:5637aa4dc8ce sp:7fff49ea3148 error:0 in 3[5637aa4d7000+7000]
7[1936330]: segfault at 8 ip 00007face8284875 sp 00007ffd59314650 error 4 in ld-2.28.so[7face8263000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[1942207]: segfault at 8 ip 00007f26a9094875 sp 00007ffd08d08210 error 4 in ld-2.28.so[7f26a9073000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[1949436]: segfault at 8 ip 00007f3dfe7e7875 sp 00007fff586d3390 error 4 in ld-2.28.so[7f3dfe7c6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[1949984]: segfault at 8 ip 00007fbd6f97a875 sp 00007ffeb7033ee0 error 4 in ld-2.28.so[7fbd6f959000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[1949893]: segfault at 8 ip 00007f5057580875 sp 00007ffe950d5680 error 4 in ld-2.28.so[7f505755f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[1954083]: segfault at 8 ip 00007fd15560c875 sp 00007ffdfbbdb300 error 4 in ld-2.28.so[7fd1555eb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[1955222]: segfault at 8 ip 00007fdb9b0c3875 sp 00007ffd0fce89d0 error 4 in ld-2.28.so[7fdb9b0a2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[1956123]: segfault at 8 ip 00007fe17aedb875 sp 00007ffffe160870 error 4 in ld-2.28.so[7fe17aeba000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[1956230]: segfault at 8 ip 00007f0d0fe94875 sp 00007ffef8bd7ce0 error 4 in ld-2.28.so[7f0d0fe73000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[1959181]: segfault at 8 ip 00007efdb0dad875 sp 00007ffc99e96870 error 4 in ld-2.28.so[7efdb0d8c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[1959370]: segfault at 8 ip 00007f687b288875 sp 00007ffd3ae95c20 error 4 in ld-2.28.so[7f687b267000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1964766:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000403:0xc83:0x0] mdc close failed: rc = -13
LustreError: 1964766:0:(file.c:251:ll_close_inode_openhandle()) Skipped 5 previous similar messages
Lustre: 1742781:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 1742781:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1745077:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
LustreError: 1967746:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000401:0x48fe:0x0] mdc close failed: rc = -13
13[1982156]: segfault at 0 ip 000055ff0e735b47 sp 00007ffea2fc7290 error 6 in 13[55ff0e731000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1742782:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
1[1985559]: segfault at 0 ip 000055a62dfe0b47 sp 00007ffe60c1f680 error 6 in 1[55a62dfdc000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1743043:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1743043:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 271 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1743043:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 277 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1743043:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 277 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1743043:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 277 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1743043:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 277 previous similar messages
Lustre: 1743043:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1743043:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 277 previous similar messages
ptlrpc_watchdog_fire: 17 callbacks suppressed
task:mdt_out00_002 state:I
Lustre: mdt_out00_004: service thread pid 1757646 was inactive for 40.542 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt_out00_000 state:I stack:0 pid:1742773 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
stack:0 pid:1745365 ppid:2 flags:0x80004080
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
Lustre: Skipped 2 previous similar messages
task:mdt_out00_004 state:I stack:0 pid:1757646 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x60/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x60/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
Call Trace:
tgt_request_handle+0x59b/0x2010 [ptlrpc]
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x60/0x2e0 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
kthread+0x1de/0x210
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? set_kthread_struct+0x70/0x70
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
ret_from_fork+0x1f/0x30
Lustre: lustre-OST0000-osc-ffff9875c514b800: disconnect after 21s idle
LustreError: 1742758:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff9875c5446800/0x347fd8e7410898c0 lrc: 3/0,0 mode: PR/PR res: [0x200000403:0x16ab:0x0].0x0 bits 0x1b/0x0 rrc: 8 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x347fd8e7410897bd expref: 1150 pid: 1757646 timeout: 9858 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 1745077:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff9876196c5080 x1875447929891456/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 1743973:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000e4c56069 ns: mdt-lustre-MDT0000_UUID lock: ffff9875c547d000/0x347fd8e741089cd3 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 26 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x347fd8e741089cc5 expref: 695 pid: 1743973 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 1743973:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) Skipped 7 previous similar messages
Lustre: mdt_out00_000: service thread pid 1742773 completed after 102.030s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9875cf553000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: Skipped 3 previous similar messages
Lustre: lustre-MDT0000-mdc-ffff9875cf553000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt_out00_002: service thread pid 1745365 completed after 102.030s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9875cf553000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt_out00_004: service thread pid 1757646 completed after 102.062s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1994527:0:(file.c:6219:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
LustreError: 1994527:0:(file.c:6219:ll_inode_revalidate_fini()) Skipped 40 previous similar messages
LustreError: 1994480:0:(mdc_request.c:1507:mdc_read_page()) lustre-MDT0000-mdc-ffff9875cf553000: [0x200000401:0x1:0x0] lock enqueue fails: rc = -5
LustreError: 1994480:0:(statahead.c:1814:is_first_dirent()) lustre: reading dir [0x200000401:0x1:0x0] at 0 stat_pid = 1994366 : rc = -5
LustreError: 1994613:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875cf553000: inode [0x200000401:0x52ee:0x0] mdc close failed: rc = -108
Lustre: lustre-MDT0000-mdc-ffff9875cf553000: Connection restored to 0@lo (at 0@lo)
18[1997119]: segfault at 8 ip 00007fbef2442875 sp 00007ffdd1e619e0 error 4 in ld-2.28.so[7fbef2421000+2f000]
10[1997090]: segfault at 8 ip 00007f1ac803b875 sp 00007ffcfd5c2960 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7f1ac801a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[1999471]: segfault at 8 ip 00007fcdea76a875 sp 00007ffc6c6cb7a0 error 4 in ld-2.28.so[7fcdea749000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[2003954]: segfault at 7fcb6a7b7130 ip 00005579d494aa64 sp 00007fffa4da9aa0 error 6 in 4[5579d4948000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 88 00 00 00 01 0f 84 cb fc ff ff 48 83 c5 01 e9 1f fa ff ff 0f
LustreError: 2008004:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875cf553000: inode [0x200000404:0x48d:0x0] mdc close failed: rc = -13
LustreError: 2008004:0:(file.c:251:ll_close_inode_openhandle()) Skipped 4 previous similar messages
19[2009226]: segfault at 8 ip 00007f1c42444875 sp 00007ffc2aa28210 error 4 in ld-2.28.so[7f1c42423000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[2010828]: segfault at 8 ip 00007f2ea17be875 sp 00007ffd9be09ce0 error 4 in ld-2.28.so[7f2ea179d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[2012128]: segfault at 8 ip 00007f7dd9c92875 sp 00007fff508ab250 error 4 in ld-2.28.so[7f7dd9c71000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[2021262]: segfault at b2000d ip 00005618bcae0230 sp 00007fd47cd0a778 error 6 in 6[5618bcada000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 2c 00 00 00 5c 01 00 00 c8 bd ff ff b1 00 00 00 <00> 45 0e 10 86 02 41 0e 18 83 03 44 0e 20 02 40 0a 0e 18 41 0e 10
11[2025460]: segfault at 1b30 ip 0000000000001b30 sp 00007ffdbb87d400 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
5[2043557]: segfault at 5585ffabe000 ip 00005585ffabe000 sp 00007ffc7ab33788 error 14 in 5[5585ffcbe000+1000]
Code: Unable to access opcode bytes at RIP 0x5585ffabdfd6.
Lustre: 1745077:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1745077:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
19[2046284]: segfault at 8 ip 00007f0347dcf875 sp 00007ffe24118830 error 4 in ld-2.28.so[7f0347dae000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[2051432]: segfault at 0 ip 000055eb5668eb47 sp 00007fffa6a34e20 error 6 in 0[55eb5668a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 2051262:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000404:0x1405:0x0] mdc close failed: rc = -13
5[2059042]: segfault at 8 ip 00007f1f5dab4875 sp 00007fff198dd6c0 error 4 in ld-2.28.so[7f1f5da93000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[2059929]: segfault at 8 ip 00007fe979574875 sp 00007ffc1e4bfba0 error 4 in ld-2.28.so[7fe979553000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[2060447]: segfault at 0 ip 000056097a04ab47 sp 00007ffd4545c0a0 error 6 in 15[56097a046000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[2061150]: segfault at 8 ip 00007f0ebd161875 sp 00007ffd512ce220 error 4 in ld-2.28.so[7f0ebd140000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[2068119]: segfault at 8 ip 00007f7f69038875 sp 00007ffd3492a020 error 4 in ld-2.28.so[7f7f69017000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[2072146]: segfault at 8 ip 00007f39da332875 sp 00007ffe88775b10 error 4 in ld-2.28.so[7f39da311000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[2088972]: segfault at 0 ip 0000556a21985a88 sp 00007ffcc360cb10 error 6 in 0[556a21984000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 14[2099029] trap invalid opcode ip:555b82ec854e sp:7fff5ed252b0 error:0 in 14[555b82ec3000+7000]
LustreError: 2099589:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000403:0x3a99:0x0] mdc close failed: rc = -13
Lustre: 1742782:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
12[2103713]: segfault at 8 ip 00007fc04a4d3875 sp 00007ffc5e59d0a0 error 4 in ld-2.28.so[7fc04a4b2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[2105289]: segfault at 0 ip 000055d15c454b47 sp 00007fff3dd7ab60 error 6 in 7[55d15c450000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5[2111317]: segfault at 1 ip 00005585edf0b950 sp 00007ffed11609d8 error 6 in 5[5585edf07000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[2111761]: segfault at 0 ip 000055ef7ebe6b47 sp 00007ffeca07cc00 error 6 in 8[55ef7ebe2000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5[2118372]: segfault at 8 ip 00007f16bed97875 sp 00007fffc7b02340 error 4 in ld-2.28.so[7f16bed76000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1789960:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 1789960:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1745393:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 525, rollback = 7
Lustre: 1745393:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 417 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 422 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745393:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 422 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/525/0, punch: 0/0/0, quota 4/78/0
Lustre: 1745393:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 422 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 422 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 422 previous similar messages
12[2135300]: segfault at 0 ip 0000563718b6eb47 sp 00007ffe3b430410 error 6 in 12[563718b6a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[2143544]: segfault at 8 ip 00007f491a683875 sp 00007ffde6933af0 error 4 in ld-2.28.so[7f491a662000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 15[2152377] general protection fault ip:5580312fa05b sp:7fffd93e3010 error:0 in 15[5580312f7000+7000]
6[2159310]: segfault at 8 ip 00007fe6cde81875 sp 00007ffd3557d410 error 4 in ld-2.28.so[7fe6cde60000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[2160485]: segfault at 0 ip 000056011a956b47 sp 00007ffc2c9f9e20 error 6 in 1[56011a952000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 3[2163641] trap invalid opcode ip:5560da1c2d72 sp:7ffebaaf7918 error:0 in 3[5560da1bc000+7000]
2[2170862]: segfault at 70 ip 00005631ec95b248 sp 00007ffce52cf648 error 4 in 2[5631ec956000+7000]
Code: 00 00 20 20 20 20 20 20 2d 2d 76 65 72 73 69 6f 6e 20 20 6f 75 74 70 75 74 20 76 65 72 73 69 6f 6e 20 69 6e 66 6f 72 6d 61 74 <69> 6f 6e 20 61 6e 64 20 65 78 69 74 0a 00 00 00 68 74 74 70 73 3a
5[2176240]: segfault at 8 ip 00007f58dc07f875 sp 00007fffa0746c90 error 4 in ld-2.28.so[7f58dc05e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[2179219]: segfault at 8 ip 00007f2b7e72c875 sp 00007ffd072d1b70 error 4 in ld-2.28.so[7f2b7e70b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[2184141]: segfault at 0 ip 00005602b0e00800 sp 00007ffc7c2be1a8 error 6 in 4[5602b0dff000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
12[2186219]: segfault at 8 ip 00007f9889c8e875 sp 00007ffff4b47dd0 error 4 in ld-2.28.so[7f9889c6d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[2189373]: segfault at 0 ip 000055dccdb40b47 sp 00007ffe4c521110 error 6 in 11[55dccdb3c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[2194386]: segfault at 8 ip 00007f206588c875 sp 00007ffeb4a00ff0 error 4 in ld-2.28.so[7f206586b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[2206055]: segfault at 1b30 ip 0000000000001b30 sp 00007ffcec102840 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
LustreError: 2208280:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9875c514b800: inode [0x200000404:0x4729:0x0] mdc close failed: rc = -13
14[2211610]: segfault at 8 ip 00007f253ec13875 sp 00007ffc812c8c10 error 4 in ld-2.28.so[7f253ebf2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[2219448]: segfault at 0 ip 0000564900e5e7a0 sp 00007ffc2098cf98 error 6 in 5[564900e5d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[2219966]: segfault at 0 ip 000055fbfeae2a88 sp 00007ffc58bb8e90 error 6 in 8[55fbfeae1000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[2223534]: segfault at 7f3060b41778 ip 000055ba210c3a21 sp 00007fffff788740 error 7 in 10[55ba210bd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 83 02 53 0a 0e 08 41 0b 10 00 00 00 58 <09> 00 00 1c dc ff ff 28 00 00 00 00 63 0e 10 10 00 00 00 6c 09 00
5[2224220]: segfault at 74 ip 0000558ee90a908d sp 00007ffea2460ac8 error 6 in 5[558ee90a4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 46 49 58 5d 2e 2e 2e 0a 20 <20> 6f 72 3a 20 20 25 73 20 4f 50 54 49 4f 4e 0a 50 61 75 73 65 20
9[2229739]: segfault at 8 ip 00007fc15e482875 sp 00007ffc25d2c210 error 4 in ld-2.28.so[7fc15e461000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[2229853]: segfault at 8 ip 00007ff0544e1875 sp 00007ffcf1d90280 error 4 in ld-2.28.so[7ff0544c0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[2231985]: segfault at 8 ip 00007f873bcc1875 sp 00007ffd664d0ac0 error 4 in ld-2.28.so[7f873bca0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[2237525]: segfault at 8 ip 00007f7e575a8875 sp 00007ffe08d223b0 error 4 in ld-2.28.so[7f7e57587000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[2242192]: segfault at ffffffffffffff90 ip 000056128e8dd72d sp 00007f025d301ccc error 7 in 0[56128e8db000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 5c 00 00 <00> 66 90 80 7c 24 42 00 0f 85 01 0f 00 00 45 31 ed 31 c0 80 7c 24
19[2247442]: segfault at 0 ip 000055a9ab8c0b47 sp 00007ffc2b2b1c70 error 6 in 19[55a9ab8bc000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[2255377]: segfault at 8 ip 00007f17335a3875 sp 00007fff2eed0f10 error 4 in ld-2.28.so[7f1733582000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 2261288:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
17[2263938]: segfault at 8 ip 00007fc31acda875 sp 00007ffd63a9a560 error 4 in ld-2.28.so[7fc31acb9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[2264353]: segfault at 8 ip 00007fe104286875 sp 00007ffe93a64760 error 4 in ld-2.28.so[7fe104265000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1742783:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1742783:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 3 previous similar messages
2[2289274]: segfault at 8 ip 00007f6c1ac13875 sp 00007ffc8e055eb0 error 4 in ld-2.28.so[7f6c1abf2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[2294292]: segfault at 0 ip 000055c582529b47 sp 00007ffcb803ec80 error 6 in 0[55c582525000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14[2294448]: segfault at 8 ip 00007f2855a8c875 sp 00007ffc79d38130 error 4 in ld-2.28.so[7f2855a6b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[2296309]: segfault at 0 ip 000055cd4315fb47 sp 00007ffc64691fc0 error 6 in 10[55cd4315b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[2300573]: segfault at 558e92c8a000 ip 0000558e92c8a000 sp 00007ffcbf9142c8 error 14 in 0[558e92e8a000+1000]
Code: Unable to access opcode bytes at RIP 0x558e92c89fd6.
17[2305576]: segfault at 1b30 ip 0000000000001b30 sp 00007ffcbc5152b0 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
4[2307039]: segfault at 0 ip 0000563b62f83b47 sp 00007ffd410f48b0 error 6 in 4[563b62f7f000+7000]
Code: Unable to access opcode bytes at RIP 0x563b62f83b1d.
16[2309464]: segfault at 0 ip 000055804445bb47 sp 00007ffeb4bb5730 error 6 in 10[558044457000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1745393:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 543, rollback = 7
Lustre: 1745393:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 513 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2092:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2092:osd_trans_dump_creds()) Skipped 521 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2099:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1745393:0:(osd_handler.c:2099:osd_trans_dump_creds()) Skipped 521 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2109:osd_trans_dump_creds()) write: 2/543/0, punch: 0/0/0, quota 4/150/0
Lustre: 1745393:0:(osd_handler.c:2109:osd_trans_dump_creds()) Skipped 521 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2116:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2116:osd_trans_dump_creds()) Skipped 521 previous similar messages
Lustre: 1745393:0:(osd_handler.c:2123:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1745393:0:(osd_handler.c:2123:osd_trans_dump_creds()) Skipped 521 previous similar messages
13[2327246]: segfault at 0 ip 000055c04c49bb47 sp 00007ffec15e2e50 error 6 in 13[55c04c497000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 2[2338843] trap invalid opcode ip:55556df787de sp:7fff4f0787f0 error:0 in 2[55556df73000+7000]
15[2345235]: segfault at 8 ip 00007fdbee8ce875 sp 00007ffe433ccb90 error 4 in ld-2.28.so[7fdbee8ad000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[2346288]: segfault at 8 ip 00007f628a0a5875 sp 00007ffeb85b6a20 error 4 in ld-2.28.so[7f628a084000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[2369822]: segfault at 8 ip 00007f93f328d875 sp 00007ffc24026b30 error 4 in ld-2.28.so[7f93f326c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[2374093]: segfault at 7ffc5e831d9a ip 00007ffc5e831d9a sp 00007ffc5e82f670 error 15
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <2f> 6d 6e 74 2f 6c 75 73 74 72 65 32 2f 72 61 63 65 72 2f 31 38 00
Lustre: 1789960:0:(osd_internal.h:1462:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 1789960:0:(osd_internal.h:1462:osd_trans_exec_op()) Skipped 5 previous similar messages
7[2391862]: segfault at 558477a3ef24 ip 0000558477a3ef24 sp 00007ffd7bc76488 error 7 in 7[558477a3a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[2405961]: segfault at 8 ip 00007f341621d875 sp 00007ffff20b0580 error 4 in ld-2.28.so[7f34161fc000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Link to test (2026-09-05 00:42)
racer test 1: racer on clients: centos-50.localnet DURATION=2700
LustreError: 1600092:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 1600092:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 2 PID: 1600092 Comm: ll_sa_1599912 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 1396940:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1396940:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1396940:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1396940:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1396940:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1396940:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1394745:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1394745:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1394745:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1394745:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1394745:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1394745:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1394745:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1394745:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1394745:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1394745:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1394745:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1394745:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1396940:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1396940:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 1396940:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1396940:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1396940:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1396940:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1396940:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1396940:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1396940:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1396940:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1396940:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1396940:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
16[1402120]: segfault at 7fff7795106b ip 0000564150155888 sp 00007fff5786cd98 error 6 in 16[56415014f000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 8e 03 42 0e 20 8d 04 42 0e 28 8c 05 44 0e 30 86 06 44 0e 38 83
Lustre: 1402317:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff970429a9e580 x1874940120300416/t4294973328(0) o101->10509fde-d81e-40d5-a7b4-5eed97df5330@0@lo:716/0 lens 376/840 e 0 to 0 dl 1788082316 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 1401439:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 603, rollback = 7
Lustre: 1401439:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1401439:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1401439:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/603/0, punch: 0/0/0, quota 4/150/0
Lustre: 1401439:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1401439:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1401439:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1398987:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1398987:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1398987:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1398987:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1398987:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1398987:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1398987:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1398987:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1398987:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1397671:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1397671:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1397671:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1397671:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1397671:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1397671:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1397671:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 17 previous similar messages
6[1417175]: segfault at 8 ip 00007fea0ada4875 sp 00007ffda161c5a0 error 4 in ld-2.28.so[7fea0ad83000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1414110:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_1413771 LIST => FNAME no wakeup.
LustreError: 1414110:0:(statahead.c:1600:ll_statahead_thread()) Skipped 1 previous similar message
6[1421794]: segfault at 8 ip 00007fc87ed02875 sp 00007fff8878d010 error 4 in ld-2.28.so[7fc87ece1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1398987:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 609, rollback = 7
Lustre: 1398987:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 33 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1398987:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/609/0, punch: 0/0/0, quota 1/3/0
Lustre: 1398987:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 33 previous similar messages
14[1424939]: segfault at 8 ip 00007f430d440875 sp 00007ffd0875a920 error 4 in ld-2.28.so[7f430d41f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[1429330]: segfault at 8 ip 00007f2500fcb875 sp 00007fff296d04d0 error 4 in ld-2.28.so[7f2500faa000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[1429940]: segfault at 8 ip 00007fc7e3567875 sp 00007ffcb0e05b60 error 4 in ld-2.28.so[7fc7e3546000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[1430185]: segfault at 8 ip 00007f09ad324875 sp 00007ffd17d78a80 error 4 in ld-2.28.so[7f09ad303000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1394484:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1397671:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1397671:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 29 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1397671:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 1397671:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 31 previous similar messages
Lustre: 1397671:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1397671:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 31 previous similar messages
7[1439368]: segfault at 0 ip 000056135e795b47 sp 00007fff431ffe60 error 6 in 7[56135e791000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[1446483]: segfault at 0 ip 000055cb55898b47 sp 00007ffd201daaa0 error 6 in 6 (deleted)[55cb55894000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7[1452516]: segfault at 8 ip 00007fa067936875 sp 00007ffc37ded360 error 4 in ld-2.28.so[7fa067915000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[1453256]: segfault at 8 ip 00007fb1261df875 sp 00007fff3bc755e0 error 4 in ld-2.28.so[7fb1261be000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1394483:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 1394483:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
18[1459103]: segfault at 0 ip 0000558ac7722b47 sp 00007ffc67edbf10 error 6 in 18[558ac771e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1394484:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1401439:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1401439:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 75 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1401439:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1401439:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1401439:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1401439:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 77 previous similar messages
Lustre: 1401439:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1401439:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 77 previous similar messages
12[1471907]: segfault at 8 ip 00007fdfbce06875 sp 00007ffce7971510 error 4 in ld-2.28.so[7fdfbcde5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[1483506]: segfault at 0 ip 00005636d55e9b47 sp 00007fff1bbb03d0 error 6 in 8[5636d55e5000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[1497156]: segfault at 0 ip 0000563019ff9b47 sp 00007ffeef47b350 error 6 in 8[563019ff5000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
17[1507388]: segfault at 8 ip 00007f8c6cafb875 sp 00007ffca3a480f0 error 4 in ld-2.28.so[7f8c6cada000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[1519780]: segfault at 5646b8bec000 ip 00005646b8bec000 sp 00007ffc3f550338 error 14 in 2[5646b8dec000+1000]
Code: Unable to access opcode bytes at RIP 0x5646b8bebfd6.
LustreError: 1521763:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9704106ee000: inode [0x200000402:0x282f:0x0] mdc close failed: rc = -13
17[1521956]: segfault at 8 ip 00007febaed60875 sp 00007fffbc0ca160 error 4 in ld-2.28.so[7febaed3f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[1527445]: segfault at 7fa108375b98 ip 000055e0373acf14 sp 00007ffc74ec38a8 error 7 in 7[55e0373a8000+7000]
Code: 5b 5d 41 5c 41 5d 41 5e 41 5f c3 66 66 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa c3 66 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa <48> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1398987:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1398987:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 155 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1398987:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1398987:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 1398987:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1398987:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 155 previous similar messages
LustreError: 1531007:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9704106ee000: inode [0x200000402:0x2b71:0x0] mdc close failed: rc = -13
4[1542886]: segfault at 8 ip 00007f3f380bf875 sp 00007ffe20cff000 error 4 in ld-2.28.so[7f3f3809e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1549913:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9704106ee000: inode [0x200000401:0x3160:0x0] mdc close failed: rc = -13
15[1550944]: segfault at 0 ip 000055fc26e0ab47 sp 00007ffc42308020 error 6 in 15[55fc26e06000+7000]
Code: Unable to access opcode bytes at RIP 0x55fc26e0ab1d.
traps: 16[1552072] general protection fault ip:55cb36df6195 sp:7ffe88804e88 error:0 in 16[55cb36df1000+7000]
2[1555262]: segfault at 0 ip 000055a9609fab47 sp 00007ffdfe1f0620 error 6 in 2[55a9609f6000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[1556191]: segfault at 8 ip 00007fe100085875 sp 00007ffffef8b600 error 4 in ld-2.28.so[7fe100064000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[1556274]: segfault at 8 ip 00007fb9654fc875 sp 00007fff7c170c30 error 4 in ld-2.28.so[7fb9654db000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[1556336]: segfault at 8 ip 00007fbb95c7f875 sp 00007ffcc3e1f240 error 4 in ld-2.28.so[7fbb95c5e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[1565959]: segfault at 8 ip 00007f38c1311875 sp 00007ffd329562d0 error 4 in ld-2.28.so[7f38c12f0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 12[1569552] trap invalid opcode ip:55e1bea534be sp:3183e281 error:0 in 12[55e1bea4e000+7000]
2[1573225]: segfault at 0 ip 00005620ccc6db47 sp 00007fffb1ca4df0 error 6 in 2[5620ccc69000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1394483:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
15[1578265]: segfault at 8 ip 00007f1b54ce5875 sp 00007ffd1a614a90 error 4 in ld-2.28.so[7f1b54cc4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[1582069]: segfault at 8 ip 00007f7440656875 sp 00007ffdfb1bbbc0 error 4 in ld-2.28.so[7f7440635000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[1584412]: segfault at 8 ip 00007fa6a0494875 sp 00007ffcbcb92520 error 4 in ld-2.28.so[7fa6a0473000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1398005:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1398005:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
3[1590445]: segfault at 0 ip 0000562a1a17fb47 sp 00007ffd7b8f50c0 error 6 in 12[562a1a17b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 1592861:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9704106ee000: inode [0x200000402:0x4047:0x0] mdc close failed: rc = -13
Link to test (2026-08-30 09:46)
racer test 1: racer on clients: centos-95.localnet DURATION=2700
LustreError: 422103:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 422103:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 9 PID: 422103 Comm: ll_sa_422084 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 5822:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9b43fb8a4280 x1874622679306752/t4294967531(0) o101->3b5e0fb2-a8e4-433e-adbe-d7e583b33ee8@0@lo:721/0 lens 376/840 e 0 to 0 dl 1787779566 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 11140:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 11140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 11140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 11140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
1[14821]: segfault at 8 ip 00007f10b1a9e875 sp 00007ffe44674180 error 4 in ld-2.28.so[7f10b1a7d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6099:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6099:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6099:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6099:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6099:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6099:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6099:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6099:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6099:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6099:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6099:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6099:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6099:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6097:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6097:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 6097:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6097:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6097:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6097:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6097:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6097:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6097:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6097:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6097:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6097:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6099:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6099:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 13 previous similar messages
Lustre: 6099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6097:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6099:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 14 previous similar messages
Lustre: 6099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6099:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 6099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6099:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 6099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6099:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 13 previous similar messages
Lustre: 6097:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 14 previous similar messages
8[27870]: segfault at 8 ip 00007f9f62143875 sp 00007ffe3b9a3970 error 4 in ld-2.28.so[7f9f62122000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 16 previous similar messages
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 17 previous similar messages
19[41348]: segfault at 0 ip 0000563e9ab63a88 sp 00007ffe2198cec0 error 6 in 19[563e9ab62000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[43239]: segfault at 8 ip 00007fd8a7801875 sp 00007ffdbee0ab90 error 4 in ld-2.28.so[7fd8a77e0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 25371:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 25371:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 29 previous similar messages
Lustre: 25371:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 25371:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 25371:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 25371:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 25371:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 25371:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 25371:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 25371:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 25371:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 25371:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 29 previous similar messages
4[46259]: segfault at 0 ip 000056159faa6b47 sp 00007ffc44c80200 error 6 in 4[56159faa2000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[46679]: segfault at 0 ip 0000562dce8a5b47 sp 00007ffccdb3aa70 error 6 in 12[562dce8a1000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
hrtimer: interrupt took 2129065 ns
16[60292]: segfault at 8 ip 00007fed8c019875 sp 00007ffd47c59660 error 4 in ld-2.28.so[7fed8bff8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 4[67533] trap invalid opcode ip:55b8f167030e sp:7fff32be3678 error:0 in 4[55b8f166a000+7000]
traps: 8[69280] trap invalid opcode ip:55ab6b59a38a sp:7ffe4aae2c48 error:0 in 8[55ab6b594000+7000]
Lustre: 8359:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 8359:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 97 previous similar messages
Lustre: 8359:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8359:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 97 previous similar messages
Lustre: 8359:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8359:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 97 previous similar messages
Lustre: 8359:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8359:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 97 previous similar messages
Lustre: 8359:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8359:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 97 previous similar messages
Lustre: 8359:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8359:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 97 previous similar messages
Lustre: 8860:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
7[90052]: segfault at 8 ip 00007f0478320875 sp 00007fff9b284fe0 error 4 in ld-2.28.so[7f04782ff000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[108466]: segfault at 7f8dc3ec4cd0 ip 0000559c11b3d690 sp 00007fffd94723d0 error 7 in 13[559c11b3a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 110198:0:(statahead.c:2416:start_statahead_thread()) lustre: invalid pattern 0X0.
11[120347]: segfault at 8 ip 00007f2430edf875 sp 00007ffd8ea3c5b0 error 4 in ld-2.28.so[7f2430ebe000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 125225:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f44af000: inode [0x200000401:0x10f2:0x0] mdc close failed: rc = -13
6[128890]: segfault at 8 ip 00007f105aa70875 sp 00007ffeb430ad10 error 4 in ld-2.28.so[7f105aa4f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 555, rollback = 7
Lustre: 11140:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 171 previous similar messages
Lustre: 11140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 172 previous similar messages
Lustre: 11140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 11140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 172 previous similar messages
Lustre: 11140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/555/0, punch: 0/0/0, quota 4/150/0
Lustre: 11140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 172 previous similar messages
Lustre: 11140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 172 previous similar messages
Lustre: 11140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 172 previous similar messages
14[129593]: segfault at fffffffff589481f ip 000055bbd87d2adc sp 00007ffd36648140 error 7 in 14[55bbd87ce000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 8b 1f 48 89 f5 48 89 44 24 08 49 89 e4 48 81 fb 00 a4 1f 00 7f
LustreError: 129593:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f44af000: inode [0x200000402:0x11de:0x0] mdc close failed: rc = -13
9[130254]: segfault at 8 ip 00007f3bc5c50875 sp 00007ffd8b735d50 error 4 in ld-2.28.so[7f3bc5c2f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[134925]: segfault at 8 ip 00007f1ddc957875 sp 00007ffc4913d5c0 error 4 in ld-2.28.so[7f1ddc936000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[140635]: segfault at 8 ip 00007fb396c76875 sp 00007ffca228ef10 error 4 in ld-2.28.so[7fb396c55000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[142773]: segfault at 8 ip 00007f2a449ef875 sp 00007ffc3707e560 error 4 in ld-2.28.so[7f2a449ce000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 9[144313] trap invalid opcode ip:55690602f54a sp:7ffd779dab50 error:0 in 9[556906029000+7000]
9[148187]: segfault at 8 ip 00007f35f1b20875 sp 00007ffe80394780 error 4 in ld-2.28.so[7f35f1aff000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[148913]: segfault at 5599e74c6000 ip 00005599e74c6000 sp 00007ffff6fb6628 error 14 in 11[5599e76c6000+1000]
Code: Unable to access opcode bytes at RIP 0x5599e74c5fd6.
10[162561]: segfault at 7fdfa1590530 ip 0000561b5516c67e sp 00007fff9c51ca90 error 6 in 10[561b55166000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00 00 b4 05 <00> 00 00 d6 ff ff 8d 00 00 00 00 48 0e 50 02 7f 0a 0e 08 41 0b 00
3[164570]: segfault at 8 ip 00007fe1e15b9875 sp 00007ffed341d4f0 error 4 in ld-2.28.so[7fe1e1598000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[166909]: segfault at 8 ip 00007f1a99805875 sp 00007ffee25ac170 error 4 in ld-2.28.so[7f1a997e4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[174950]: segfault at 8 ip 00007f5e74acb875 sp 00007fff9076f6e0 error 4 in ld-2.28.so[7f5e74aaa000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[175420]: segfault at 8 ip 00007f7bbd18d875 sp 00007ffe54f84ca0 error 4 in ld-2.28.so[7f7bbd16c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[176996]: segfault at 8 ip 00007f54c5c0c875 sp 00007fffaec5d3d0 error 4 in ld-2.28.so[7f54c5beb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 182769:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f44af000: inode [0x200000402:0x18fe:0x0] mdc close failed: rc = -13
Lustre: 8860:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
6[187219]: segfault at 7fff81f99f4e ip 00005570cda559f2 sp 00007fff40a45d30 error 6 in 6[5570cda52000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 89 fe 41 55 41 54 49 89 d4 55 53 48 89 cb 48 83 ec 38 48 85 c9
12[196083]: segfault at 8 ip 00007fe450cfb875 sp 00007ffda5cb6f30 error 4 in ld-2.28.so[7fe450cda000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[196360]: segfault at 8 ip 00007f9d0bd99875 sp 00007ffce3343bb0 error 4 in ld-2.28.so[7f9d0bd78000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[208300]: segfault at 55f6373a7000 ip 000055f6373a7000 sp 00007fffd65124d8 error 14 in 3[55f6375a7000+1000]
Code: Unable to access opcode bytes at RIP 0x55f6373a6fd6.
13[225246]: segfault at 8 ip 00007fbef5571875 sp 00007ffe54307200 error 4 in ld-2.28.so[7fbef5550000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 5835:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 5836:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 5836:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 89613:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 89613:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
traps: 3[243151] trap invalid opcode ip:561309afb8ce sp:7fffd4bb5160 error:0 in 3[561309af5000+7000]
12[244122]: segfault at 0 ip 000056346aeacb47 sp 00007ffcf0939130 error 6 in 12[56346aea8000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4[245135]: segfault at 8 ip 00007fd2f82b2875 sp 00007ffe9b710690 error 4 in ld-2.28.so[7fd2f8291000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11140:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 11140:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 309 previous similar messages
Lustre: 11140:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11140:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 316 previous similar messages
Lustre: 11140:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 11140:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 316 previous similar messages
Lustre: 11140:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 11140:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 316 previous similar messages
Lustre: 11140:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11140:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 316 previous similar messages
Lustre: 11140:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11140:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 316 previous similar messages
14[250057]: segfault at 8 ip 00007f28c314e875 sp 00007ffe0096ca50 error 4 in ld-2.28.so[7f28c312d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[252118]: segfault at 8 ip 00007f518b388875 sp 00007ffee1f1b6c0 error 4 in ld-2.28.so[7f518b367000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[256559]: segfault at 1 ip 0000558d320a0950 sp 00007ffd61f39e88 error 6 in 2[558d3209c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[275987]: segfault at 8 ip 00007fd77c116875 sp 00007ffd1218f210 error 4 in ld-2.28.so[7fd77c0f5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[276137]: segfault at 8 ip 00007f3685114875 sp 00007ffc0077b8b0 error 4 in ld-2.28.so[7f36850f3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[282441]: segfault at 8 ip 00007fab1ca70875 sp 00007ffe4dcaa090 error 4 in ld-2.28.so[7fab1ca4f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[282237]: segfault at 0 ip 00005646c3907b47 sp 00007fff7ea73880 error 6 in 16[5646c3903000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 288748:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f4aed000: inode [0x200000401:0x2550:0x0] mdc close failed: rc = -13
LustreError: 293642:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f44af000: inode [0x200000401:0x26b3:0x0] mdc close failed: rc = -13
15[297249]: segfault at 8 ip 00007fbac08ca875 sp 00007ffc1f2d7360 error 4 in ld-2.28.so[7fbac08a9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[298175]: segfault at 8 ip 00007f8ee6ee8875 sp 00007ffd41ff69a0 error 4 in ld-2.28.so[7f8ee6ec7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[301094]: segfault at 0 ip 00005580d482e200 sp 00007fffefbdb678 error 6 in 1[5580d482c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14[311767]: segfault at 8 ip 00007f0514092875 sp 00007ffea26e8be0 error 4 in ld-2.28.so[7f0514071000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[312481]: segfault at 8 ip 00007f3bcebf2875 sp 00007ffc6bfa45c0 error 4 in ld-2.28.so[7f3bcebd1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 312455:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f4aed000: inode [0x200000402:0x2a62:0x0] mdc close failed: rc = -13
13[322986]: segfault at 5607de2bd000 ip 00005607de2bd000 sp 00007ffe271ee858 error 14 in 13[5607de4bd000+1000]
Code: Unable to access opcode bytes at RIP 0x5607de2bcfd6.
Lustre: 5835:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 5835:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
12[326864]: segfault at 8 ip 00007fbf975fa875 sp 00007fffec4b6650 error 4 in ld-2.28.so[7fbf975d9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[331270]: segfault at 8 ip 00007fe37ab2c875 sp 00007ffc618ae230 error 4 in ld-2.28.so[7fe37ab0b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[331485]: segfault at 8 ip 00007f536c05c875 sp 00007ffd95099970 error 4 in ld-2.28.so[7f536c03b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 3[332724] trap invalid opcode ip:564c142f2716 sp:7ffebafc6a58 error:0 in 3[564c142ed000+7000]
18[333973]: segfault at 8 ip 00007efc5bfc2875 sp 00007fffceb0e4f0 error 4 in ld-2.28.so[7efc5bfa1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: mdt00_010: service thread pid 8313 was inactive for 41.879 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: mdt00_004: service thread pid 7884 was inactive for 41.869 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_out00_00 state:I
task:mdt00_006 state:I stack:0 pid:8107 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
Lustre: Skipped 2 previous similar messages
task:mdt00_004 state:I stack:0 pid:7884 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
stack:0 pid:6103 ppid:2 flags:0x80004080
Call Trace:
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
? do_raw_spin_unlock+0x79/0x190
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
? woken_wake_function+0x40/0x40
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? lu_name_is_valid_len+0x65/0x80 [mdt]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_reply_header_read.isra.14+0x1d0/0x1d0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 5810:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff9b43ff1d5200/0x3122329a614b7957 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x2d9e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x3122329a614b78e7 expref: 861 pid: 6103 timeout: 1660 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt_io00_004: service thread pid 89613 completed after 103.843s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 5821 completed after 103.800s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 8107 completed after 103.804s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 6653 completed after 103.819s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_001: service thread pid 5822 completed after 103.823s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_013: service thread pid 10580 completed after 103.815s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 6103:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000004c6b4dbe ns: mdt-lustre-MDT0000_UUID lock: ffff9b4402ec2800/0x3122329a614b7e12 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x2d9e:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x3122329a614b790a expref: 467 pid: 6103 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: ll_ost_out00_00: service thread pid 6103 completed after 103.766s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_016: service thread pid 11136 completed after 103.819s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 111724 completed after 103.819s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_005: service thread pid 7913 completed after 103.814s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_022: service thread pid 333280 completed after 103.822s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_002: service thread pid 5823 completed after 103.349s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9b43f44af000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff9b43f44af000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_008: service thread pid 8300 completed after 103.330s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 8309 completed after 103.358s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_012: service thread pid 8725 completed after 103.316s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_007: service thread pid 8147 completed after 103.357s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 8313 completed after 103.358s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 8320 completed after 103.357s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9b43f44af000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 334709:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -107
Lustre: mdt00_004: service thread pid 7884 completed after 103.365s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 334757:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f44af000: inode [0x200000401:0x1:0x0] mdc close failed: rc = -108
Lustre: lustre-MDT0000-mdc-ffff9b43f44af000: Connection restored to 0@lo (at 0@lo)
4[338454]: segfault at 8 ip 00007fab67769875 sp 00007ffdd0ce0190 error 4 in ld-2.28.so[7fab67748000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[340326]: segfault at 55e82c5edf24 ip 000055e82c5edf30 sp 00007ffef77623b8 error 7 in 14[55e82c5e9000+7000]
Code: 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 8b 15 45 2c 20 00 31 f6 e9 ae c8 ff ff 00 00 f3 0f 1e fa 48 83 ec 08 48 83 c4 08 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 5834:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
traps: 15[342602] trap invalid opcode ip:55bde08946f2 sp:7ffc7f6a1d58 error:0 in 15[55bde088f000+7000]
traps: 15[342813] trap invalid opcode ip:55bb103426f2 sp:7ffd6823a888 error:0 in 15[55bb1033d000+7000]
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6098:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 317 previous similar messages
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6098:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 319 previous similar messages
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6098:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 319 previous similar messages
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6098:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 319 previous similar messages
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6098:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 319 previous similar messages
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6098:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 319 previous similar messages
10[376526]: segfault at 8 ip 00007f345fb20875 sp 00007fff07f30120 error 4 in ld-2.28.so[7f345faff000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[376731]: segfault at 8 ip 00007f6161d79875 sp 00007ffc79f6c840 error 4 in ld-2.28.so[7f6161d58000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[385989]: segfault at 74 ip 000055f54203ebae sp 00007fff3e191798 error 4 in 11[55f542039000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <69> 73 74 72 69 62 75 74 65 20 69 74 2e 0a 54 68 65 72 65 20 69 73
0[392374]: segfault at 0 ip 0000556bc5220920 sp 00007fffec09dc00 error 6 in 0[556bc521c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
12[395655]: segfault at 0 ip 000055614a1fbb47 sp 00007ffe8532fb50 error 6 in 12[55614a1f7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[400391]: segfault at 8 ip 00007f513f5b8875 sp 00007ffd00d798e0 error 4 in ld-2.28.so[7f513f597000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[400550]: segfault at 8 ip 00007fd9eb152875 sp 00007fff4125c890 error 4 in ld-2.28.so[7fd9eb131000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 401810:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9b43f4aed000: inode [0x200000402:0x4631:0x0] mdc close failed: rc = -13
LustreError: 401810:0:(file.c:251:ll_close_inode_openhandle()) Skipped 6 previous similar messages
Lustre: 5836:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 520, rollback = 2
8[412647]: segfault at 8 ip 00007ff1bacdb875 sp 00007ffcc6917a60 error 4 in ld-2.28.so[7ff1bacba000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[413641]: segfault at 0 ip 0000562332a6bb47 sp 00007fffa1c02fc0 error 6 in 3[562332a67000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 7[420061] trap invalid opcode ip:55bb2c414986 sp:7ffffc219118 error:0 in 7[55bb2c40f000+7000]
Link to test (2026-08-26 21:58)
racer test 1: racer on clients: centos-120.localnet DURATION=2700
LustreError: 3427982:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 3427982:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 4 PID: 3427982 Comm: ll_sa_3427959 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 3116389:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3116389:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3116389:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3116389:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
12[3119500]: segfault at 8 ip 00007fa485ec1875 sp 00007ffe53930fd0 error 4 in ld-2.28.so[7fa485ea0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3116104:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff8e25dbeb3800 x1874541868808704/t4294970902(0) o101->4fcc64a7-4b25-4bb6-9661-3b53e5b7c5aa@0@lo:673/0 lens 376/864 e 0 to 0 dl 1787702508 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 3116390:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3116390:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3116390:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3116390:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116390:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3116390:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116390:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3116390:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116390:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3116390:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116390:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3116390:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116118:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3116118:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 3116118:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116118:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0
Lustre: 3116118:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116118:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 7/249/0
Lustre: 3116118:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116118:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 7/117/1, delete: 3/6/1
Lustre: 3116118:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3116118:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 3116118:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3118378:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3118378:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3118378:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3118378:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3118378:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3118378:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3118378:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3118378:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3118378:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3118378:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3118378:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3118378:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3118378:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
2[3127573]: segfault at 7ffcfdc17ff8 ip 0000561f9c013c52 sp 00007ffcfdc18000 error 6 in 2 (deleted)[561f9c010000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 24 20 00 00 00 00 48 c7 44 24 28 00 00 00 00 48 c7 44 24 30 00 00 00 00 <e8> 39 fa ff ff 48 8b 4c 24 38 64 48 33 0c 25 28 00 00 00 75 05 48
Lustre: mdt00_021: service thread pid 3128356 was inactive for 41.689 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt00_004 state:I
task:mdt00_012 state:I stack:0 pid:3119141 ppid:2 flags:0x80004080
Call Trace:
Lustre: mdt00_008: service thread pid 3118298 was inactive for 42.283 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
Lustre: Skipped 2 previous similar messages
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
stack:0 pid:3117323 ppid:2 flags:0x80004080
Call Trace:
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
task:mdt00_021 state:I
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
__schedule+0x351/0xcb0
stack:0 pid:3128356 ppid:2 flags:0x80004080
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
Call Trace:
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
__schedule+0x351/0xcb0
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
schedule+0xc4/0x180
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
schedule_timeout+0xb1/0x190
tgt_enqueue+0xd7/0x320 [ptlrpc]
? __next_timer_interrupt+0x170/0x170
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 3116093:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff8e265d73da00/0xaaf2551d0901bf3e lrc: 3/0,0 mode: PR/PR res: [0x200000402:0xd:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xaaf2551d0901b3ad expref: 176 pid: 3116108 timeout: 57455 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_007: service thread pid 3118272 completed after 100.224s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_001: service thread pid 3116104 completed after 100.289s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_006: service thread pid 3118270 completed after 100.706s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_001: service thread pid 3116117 completed after 100.618s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 3118298:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 0000000045d0bbe9 ns: mdt-lustre-MDT0000_UUID lock: ffff8e264c39fc00/0xaaf2551d0901caeb lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 22 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xaaf2551d0901cadd expref: 24 pid: 3118298 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_008: service thread pid 3118298 completed after 99.641s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_021: service thread pid 3128356 completed after 99.048s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_019: service thread pid 3126105 completed after 99.655s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_020: service thread pid 3128353 completed after 99.063s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 3116661 completed after 99.651s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_018: service thread pid 3122862 completed after 99.641s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_012: service thread pid 3119141 completed after 99.653s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 3120898 completed after 99.660s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8e25c96c9800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-MDT0000-mdc-ffff8e25c96c9800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_015: service thread pid 3119153 completed after 99.640s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 3119152 completed after 100.324s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_000: service thread pid 3116103 completed after 99.663s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_000: service thread pid 3116108 completed after 100.738s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 3117323 completed after 100.287s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 3118334 completed after 100.080s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_016: service thread pid 3119431 completed after 100.343s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 3118308 completed after 100.264s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_011: service thread pid 3119029 completed after 100.334s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff8e25c96c9800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 3128339:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
LustreError: 3128192:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e25c96c9800: inode [0x200000402:0x265:0x0] mdc close failed: rc = -5
Lustre: mdt00_005: service thread pid 3118224 completed after 100.137s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff8e25c96c9800: Connection restored to 0@lo (at 0@lo)
Lustre: 3123470:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3123470:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3123470:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3123470:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3123470:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3123470:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3123470:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3123470:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3123470:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3123470:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3123470:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3123470:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3119628:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3119628:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
9[3132672]: segfault at 8 ip 00007f1ac3ac0875 sp 00007ffdad8b9870 error 4 in ld-2.28.so[7f1ac3a9f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3116390:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3116390:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 3116390:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3116390:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 3116390:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3116390:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 3116390:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3116390:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 3116390:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3116390:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages
10[3134402]: segfault at ffffffffc5894820 ip 000055cca5000d84 sp 00007ffd89fe2420 error 7 in 10[55cca4ffc000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 8b 20 48 89 c5 e8 91 c8 ff ff 45 85 e4 75 3c 5b 5d 41 5c c3 0f
15[3136754]: segfault at 0 ip 000055854cf35b47 sp 00007ffc2de8f9e0 error 6 in 15[55854cf31000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 3123470:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 3123470:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 23 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 35 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3116389:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 35 previous similar messages
Lustre: 3116390:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3116389:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 35 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 35 previous similar messages
Lustre: 3116390:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 36 previous similar messages
Lustre: 3116117:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3116117:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 3144809:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e25c96c9800: inode [0x200000403:0x502:0x0] mdc close failed: rc = -13
LustreError: 3144809:0:(file.c:251:ll_close_inode_openhandle()) Skipped 6 previous similar messages
Lustre: 3118378:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3118378:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 27 previous similar messages
10[3152262]: segfault at 8 ip 00007f180fed7875 sp 00007ffc7a596950 error 4 in ld-2.28.so[7f180feb6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[3154185]: segfault at 8 ip 00007f8a0386c875 sp 00007ffedbcad760 error 4 in ld-2.28.so[7f8a0384b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3116388:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3116388:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 49 previous similar messages
Lustre: 3116388:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3116388:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 3116388:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3116388:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 60 previous similar messages
Lustre: 3116388:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 3116388:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 60 previous similar messages
Lustre: 3116388:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3116388:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 3116388:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3116388:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 61 previous similar messages
14[3168952]: segfault at 8 ip 00007fdda763c875 sp 00007ffcda192750 error 4 in ld-2.28.so[7fdda761b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 3172595:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e25c96c9800: inode [0x200000403:0xd8e:0x0] mdc close failed: rc = -13
10[3176714]: segfault at 0 ip 00005581d0d82b47 sp 00007ffe98b30950 error 6 in 10[5581d0d7e000+7000]
Code: Unable to access opcode bytes at RIP 0x5581d0d82b1d.
1[3178489]: segfault at 8 ip 00007fbdf903b875 sp 00007fff18efea10 error 4 in ld-2.28.so[7fbdf901a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3123470:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3123470:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 89 previous similar messages
Lustre: 3123470:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3123470:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 3123470:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3123470:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 3123470:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3123470:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 3123470:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3123470:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 3123470:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3123470:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 89 previous similar messages
Lustre: 3116117:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 3116117:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
10[3211809]: segfault at 55ca52673e36 ip 000055c9e0211dcc sp 00007ffd8af4e9b8 error 6 in 10[55c9e020c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 72 69 67 68 74 20 25 73 <20> 25 64 20 46 72 65 65 20 53 6f 66 74 77 61 72 65 20 46 6f 75 6e
traps: 7[3215421] trap invalid opcode ip:55aa260ef736 sp:7ffe258ff968 error:0 in 7[55aa260ea000+7000]
Lustre: 3116118:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
12[3216577]: segfault at 10 ip 0000562e48d444f0 sp 00007ffd8b935cd0 error 6 in 12[562e48d3e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 46 0e 10 8f 02 4c 0e 18 8e 03 45 0e 20 8d 04 45 0e 28 8c 05 41
Lustre: 3118530:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3118530:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3116116:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 3116116:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
17[3239828]: segfault at 8 ip 00007fed0b743875 sp 00007fffc5f782f0 error 4 in ld-2.28.so[7fed0b722000+2f000]
17[3239838]: segfault at 8 ip 00007f17cdd0f875 sp 00007fff3a92c4d0 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7f17cdcee000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3116117:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 3116117:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
18[3249837]: segfault at 0 ip 00005566411afb47 sp 00007ffc811b9c70 error 6 in 18[5566411ab000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1[3252363]: segfault at 0 ip 000055920b605b47 sp 00007ffd3eab73b0 error 6 in 1[55920b601000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 3256437:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff8e25c96c9800: inode [0x200000403:0x295b:0x0] mdc close failed: rc = -13
Lustre: 3116389:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3116389:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 147 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3116389:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3116389:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 155 previous similar messages
Lustre: 3116389:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3116389:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 155 previous similar messages
7[3261811]: segfault at 8 ip 00007fa7dd6ec875 sp 00007fffa606e050 error 4 in ld-2.28.so[7fa7dd6cb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 13[3264995] trap invalid opcode ip:56526c06091a sp:7ffe71651698 error:0 in 13[56526c05a000+7000]
15[3266606]: segfault at 8 ip 00007f31a0d85875 sp 00007ffd4fb77bb0 error 4 in ld-2.28.so[7f31a0d64000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[3271442]: segfault at 8 ip 00007f3da8fd5875 sp 00007fff47bc9480 error 4 in ld-2.28.so[7f3da8fb4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[3271784]: segfault at 8 ip 00007ff1a9b4e875 sp 00007ffe32aafa60 error 4 in ld-2.28.so[7ff1a9b2d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 7[3272794] trap invalid opcode ip:55e70e84aa4e sp:7fff9276bc98 error:0 in 7[55e70e844000+7000]
12[3276497]: segfault at 557761527000 ip 0000557761527000 sp 00007fff93a5e2b0 error 14 in 12[557761727000+1000]
Code: Unable to access opcode bytes at RIP 0x557761526fd6.
5[3280357]: segfault at 8 ip 00007f319fe1e875 sp 00007fffa853dcf0 error 4 in ld-2.28.so[7f319fdfd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 14[3284817] general protection fault ip:559687ce6b94 sp:7ffc0edb80a0 error:0 in 14[559687ce3000+7000]
9[3297195]: segfault at 8 ip 00007fcfa1e09875 sp 00007fffe65555d0 error 4 in ld-2.28.so[7fcfa1de8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[3299902]: segfault at 8 ip 00007f6281499875 sp 00007ffe5b1bbad0 error 4 in ld-2.28.so[7f6281478000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[3301773]: segfault at 8 ip 00007ffa5d3cc875 sp 00007ffe2f540410 error 4 in ld-2.28.so[7ffa5d3ab000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[3304958]: segfault at 8 ip 00007f7417a38875 sp 00007ffdc8f061b0 error 4 in ld-2.28.so[7f7417a17000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[3305684]: segfault at 0 ip 000055d69b9d2b47 sp 00007ffc4d359b70 error 6 in 0[55d69b9ce000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18[3307860]: segfault at 8 ip 00007f88a8429875 sp 00007ffcf8783180 error 4 in ld-2.28.so[7f88a8408000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[3311499]: segfault at 0 ip 0000558473c61b47 sp 00007ffcb22a1350 error 6 in 0[558473c5d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[3326855]: segfault at 0 ip 0000560374e91b47 sp 00007fff9ed4a520 error 6 in 8[560374e8d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 3183634:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
10[3340931]: segfault at 8 ip 00007f561082f875 sp 00007ffec6a309a0 error 4 in ld-2.28.so[7f561080e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[3341808]: segfault at 8 ip 00007f04146d9875 sp 00007ffffc0ff090 error 4 in ld-2.28.so[7f04146b8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[3341957]: segfault at 8 ip 00007ff2b037a875 sp 00007ffcc3e62cc0 error 4 in ld-2.28.so[7ff2b0359000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[3342043]: segfault at 8 ip 00007fcaab60e875 sp 00007fff386c0790 error 4 in ld-2.28.so[7fcaab5ed000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[3350474]: segfault at 0 ip 000055e95fe3a200 sp 00007ffd93a06e18 error 6 in 19[55e95fe38000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7[3369290]: segfault at 8 ip 00007f2770f90875 sp 00007ffeff16eb70 error 4 in ld-2.28.so[7f2770f6f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[3373254]: segfault at 0 ip 000055c715063b47 sp 00007fff7f2cf560 error 6 in 12[55c71505f000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[3380046]: segfault at 0 ip 000055d96970eb47 sp 00007fff927cba40 error 6 in 16[55d96970a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9[3384629]: segfault at 8 ip 00007f2c8b38b875 sp 00007ffff5f996e0 error 4 in ld-2.28.so[7f2c8b36a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3118491:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3118491:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 327 previous similar messages
Lustre: 3118491:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3118491:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 330 previous similar messages
Lustre: 3118491:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3118491:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 330 previous similar messages
Lustre: 3118491:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3118491:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 330 previous similar messages
Lustre: 3118491:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3118491:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 330 previous similar messages
Lustre: 3118491:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3118491:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 330 previous similar messages
8[3391003]: segfault at 8 ip 00007f5a6bddb875 sp 00007ffe3aa014b0 error 4 in ld-2.28.so[7f5a6bdba000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[3392879]: segfault at 8 ip 00007f0948fce875 sp 00007ffd503f3db0 error 4 in ld-2.28.so[7f0948fad000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[3395095]: segfault at 8 ip 00007fac3892a875 sp 00007ffcbba69820 error 4 in ld-2.28.so[7fac38909000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[3398812]: segfault at 8 ip 00007fb03a650875 sp 00007ffcd920bbb0 error 4 in ld-2.28.so[7fb03a62f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[3399148]: segfault at 8 ip 00007f33b2bd7875 sp 00007ffce9249650 error 4 in ld-2.28.so[7f33b2bb6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[3399732]: segfault at 8 ip 00007fef2220c875 sp 00007ffe26698d20 error 4 in ld-2.28.so[7fef221eb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3116118:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3116118:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
2[3411726]: segfault at 8 ip 00007fb242728875 sp 00007fff7fd96d10 error 4 in ld-2.28.so[7fb242707000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[3418177]: segfault at 8 ip 00007f330ea2f875 sp 00007ffc9b034df0 error 4 in ld-2.28.so[7f330ea0e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[3431141]: segfault at 8 ip 00007ff0c5ac2875 sp 00007ffd97434730 error 4 in ld-2.28.so[7ff0c5aa1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Link to test (2026-08-26 00:24)
racer test 1: racer on clients: centos-30.localnet DURATION=2700
LustreError: 232951:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 232951:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 11 PID: 232951 Comm: ll_sa_226643 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7553:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
3[12486]: segfault at 8 ip 00007f9fafa92875 sp 00007fffddf9c730 error 4 in ld-2.28.so[7f9fafa71000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[15239]: segfault at 8 ip 00007fb66a44d875 sp 00007ffcbab73340 error 4 in ld-2.28.so[7fb66a42c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 10099:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10099:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10099:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10099:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10099:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10099:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10099:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 9935:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa0bfd2fd8700 x1874504172541184/t4294969324(0) o101->06a966ab-c66b-447d-82e1-56b38ee3932c@0@lo:247/0 lens 376/864 e 0 to 0 dl 1787666597 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 12086:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa0bf4938d780 x1874504174320384/t4294971878(0) o101->06a966ab-c66b-447d-82e1-56b38ee3932c@0@lo:257/0 lens 376/840 e 0 to 0 dl 1787666607 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 14207:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 14207:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 14207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 14207:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14207:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 14207:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14207:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 14207:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14207:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 14207:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 14207:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 14207:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 13373:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_11421 LIST => FNAME no wakeup.
7[23199]: segfault at 0 ip 000055a99352d3a0 sp 00007ffc69f963e8 error 6 in 7[55a99352b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 10082:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa0bf7612c600 x1874504178865920/t4294972628(0) o101->06a966ab-c66b-447d-82e1-56b38ee3932c@0@lo:284/0 lens 376/864 e 0 to 0 dl 1787666634 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
19[25174]: segfault at 5616884a3000 ip 00007ff73a24ce8b sp 00007ffe58b3bd90 error 6 in ld-2.28.so[7ff73a23a000+2f000]
Code: 48 8d 35 b8 26 01 00 4a 63 04 a6 48 01 f0 3e ff e0 66 0f 1f 44 00 00 48 8b 45 88 4c 8b 48 10 48 8b 85 70 ff ff ff 4c 03 4b 10 <4c> 89 08 8b 43 08 48 83 f8 07 0f 84 a6 01 00 00 48 83 c3 18 48 39
6[25362]: segfault at 8 ip 00007f5c53eb8875 sp 00007fffcd49ed70 error 4 in ld-2.28.so[7f5c53e97000+2f000]
6[25749]: segfault at 8 ip 00007f4b94e31875 sp 00007ffd3780a410 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7f4b94e10000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 14207:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 4 previous similar messages
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 14207:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10099:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 10099:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
Lustre: 10099:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10099:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10099:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10099:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10099:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10099:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10099:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10099:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10099:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10099:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 17329:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 585, rollback = 7
Lustre: 17329:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 17329:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 17329:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17329:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 17329:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17329:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/585/0, punch: 0/0/0, quota 1/3/0
Lustre: 17329:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17329:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 17329:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 17329:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17329:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
17[30732]: segfault at 8 ip 00007f5eecb19875 sp 00007ffca9c7ffe0 error 4 in ld-2.28.so[7f5eecaf8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 17329:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 17329:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages
Lustre: 17329:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 17329:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 17329:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 17329:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 17329:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 17329:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 17329:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 17329:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 17329:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 17329:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 33450:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
LustreError: 39788:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0bf57cc7000: inode [0x200000403:0x40a:0x0] mdc close failed: rc = -13
Lustre: 29367:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 579, rollback = 7
Lustre: 29367:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 13 previous similar messages
Lustre: 29367:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 29367:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 29367:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 29367:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 29367:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/579/0, punch: 0/0/0, quota 1/3/0
Lustre: 29367:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 29367:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 29367:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 29367:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 29367:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 15 previous similar messages
12[50448]: segfault at 8 ip 00007fbd62e87875 sp 00007ffc986a6f90 error 4 in ld-2.28.so[7fbd62e66000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[50795]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[50795]: Failed to get EXE, ignoring: No such process
Lustre: 6777:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 6777:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 55831:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0bf57cc7000: inode [0x240000404:0x344:0x0] mdc close failed: rc = -13
5[55819]: segfault at 8 ip 00007f05aa456875 sp 00007ffe01c51c60 error 4 in ld-2.28.so[7f05aa435000+2f000]
5[55728]: segfault at 8 ip 00007f58c1ee7875 sp 00007fffdcaeecd0 error 4 in ld-2.28.so[7f58c1ec6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[56598]: Failed to get EXE, ignoring: No such process
systemd-coredump[56667]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[56667]: Failed to get EXE, ignoring: No such process
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 19 previous similar messages
Lustre: 7553:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7553:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 20 previous similar messages
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 20 previous similar messages
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 20 previous similar messages
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 20 previous similar messages
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 20 previous similar messages
13[62081]: segfault at 8 ip 00007f5b4fdb9875 sp 00007ffdf40dc5f0 error 4 in ld-2.28.so[7f5b4fd98000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 65780:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0bf5911e000: inode [0x200000404:0x85b:0x0] mdc close failed: rc = -13
18[81541]: segfault at 0 ip 00005613bb7d3b47 sp 00007ffd74959e70 error 6 in 18[5613bb7cf000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 7555:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7555:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 49 previous similar messages
Lustre: 7555:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7555:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 7555:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7555:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 7555:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7555:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 7555:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7555:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 49 previous similar messages
Lustre: 7555:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7555:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 49 previous similar messages
systemd-coredump[82059]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[82059]: Failed to get EXE, ignoring: No such process
2[90830]: segfault at 8 ip 00007f67df43e875 sp 00007fffd1e24c40 error 4 in ld-2.28.so[7f67df41d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[95371]: segfault at 8 ip 00007fa73dc1e875 sp 00007ffda2f1acf0 error 4 in ld-2.28.so[7fa73dbfd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[96118]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[96118]: Failed to get EXE, ignoring: No such process
9[100530]: segfault at 0 ip 0000560287dbeb47 sp 00007ffd40af6a00 error 6 in 9[560287dba000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
systemd-coredump[100969]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[100969]: Failed to get EXE, ignoring: No such process
17[102113]: segfault at 8 ip 00007f39ac52d875 sp 00007fff264824d0 error 4 in ld-2.28.so[7f39ac50c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[103951]: segfault at 8 ip 00007f20fd753875 sp 00007fff4b9bab50 error 4 in ld-2.28.so[7f20fd732000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[104028]: segfault at 8 ip 00007fc1accd6875 sp 00007ffea6af70b0 error 4 in ld-2.28.so[7fc1accb5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[104127]: segfault at 8 ip 00007f8413497875 sp 00007ffe2ca143a0 error 4 in ld-2.28.so[7f8413476000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[104735]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[104735]: Failed to get EXE, ignoring: No such process
systemd-coredump[104693]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[104693]: Failed to get EXE, ignoring: No such process
2[107867]: segfault at 8 ip 00007f0f1af30875 sp 00007ffe27931cd0 error 4 in ld-2.28.so[7f0f1af0f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[108588]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[108588]: Failed to get EXE, ignoring: No such process
Lustre: 13713:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 513 < left 520, rollback = 2
15[124714]: segfault at 8 ip 00007f0eb9936875 sp 00007ffd9dee1e80 error 4 in ld-2.28.so[7f0eb9915000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[125409]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[125409]: Failed to get EXE, ignoring: No such process
19[128559]: segfault at 800 ip 000055ddcbccfd3e sp 00007fff9b8e29c8 error 6 in 19[55ddcbcca000+7000]
Code: 73 2c 0a 25 73 2c 20 25 73 2c 20 25 73 2c 20 25 73 2c 0a 25 73 2c 20 25 73 2c 20 61 6e 64 20 6f 74 68 65 72 73 2e 0a 00 00 00 <00> 00 47 65 6e 65 72 61 6c 20 68 65 6c 70 20 75 73 69 6e 67 20 47
16[133425]: segfault at 8 ip 00007fef822c8875 sp 00007fff0d023620 error 4 in ld-2.28.so[7fef822a7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[134045]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[134045]: Failed to get EXE, ignoring: No such process
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 543, rollback = 7
Lustre: 7553:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 143 previous similar messages
Lustre: 7553:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7553:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 144 previous similar messages
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7553:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 144 previous similar messages
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/543/0, punch: 0/0/0, quota 4/150/0
Lustre: 7553:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 144 previous similar messages
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7553:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 144 previous similar messages
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7553:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 144 previous similar messages
8[137310]: segfault at 8 ip 00007f56b8777875 sp 00007ffc75b98b30 error 4 in ld-2.28.so[7f56b8756000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[137362]: segfault at 8 ip 00007fbb05d13875 sp 00007ffc128c0850 error 4 in ld-2.28.so[7fbb05cf2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[137636]: Failed to get EXE, ignoring: No such process
12[147710]: segfault at 8 ip 00007f5adf340875 sp 00007ffdf71cc660 error 4 in ld-2.28.so[7f5adf31f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[148307]: segfault at 8 ip 00007fb079fc2875 sp 00007ffe49fd7a30 error 4 in ld-2.28.so[7fb079fa1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[148984]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[148984]: Failed to get EXE, ignoring: No such process
10[153165]: segfault at 8 ip 00007f29e959d875 sp 00007fffaf078ed0 error 4 in ld-2.28.so[7f29e957c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[153584]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[153584]: Failed to get EXE, ignoring: No such process
10[154203]: segfault at 1 ip 00005574f8bda950 sp 00007ffc82aa4f48 error 6 in 10[5574f8bd6000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[157156]: segfault at 8 ip 00007fa39ddd3875 sp 00007ffcde15ea90 error 4 in ld-2.28.so[7fa39ddb2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[158872]: segfault at 0 ip 000055d950f07b47 sp 00007ffc1ef33910 error 6 in 10[55d950f03000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 10[159232] general protection fault ip:55bb18a2cff2 sp:7ffc9da35b98 error:0 in 10[55bb18a28000+7000]
11[160791]: segfault at 8 ip 00007f2ccfd44875 sp 00007fff5a09f6d0 error 4 in ld-2.28.so[7f2ccfd23000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[161127]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[161127]: Failed to get EXE, ignoring: No such process
17[163248]: segfault at 8 ip 00007fdac09fd875 sp 00007ffc4361c490 error 4 in ld-2.28.so[7fdac09dc000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[165490]: segfault at 2 ip 0000558a273875fc sp 00007ffe6b0737c8 error 6 in 9[558a27381000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <a4> d5 ff ff 10 00 00 00 00 00 00 00 10 00 00 00 44 05 00 00 a0 d5
7[167416]: segfault at 8 ip 00007f9a77d33875 sp 00007ffffaec2300 error 4 in ld-2.28.so[7f9a77d12000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[168225]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[168225]: Failed to get EXE, ignoring: No such process
systemd-coredump[176398]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[176398]: Failed to get EXE, ignoring: No such process
Lustre: 6777:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
8[178128]: segfault at 8 ip 00007fc9904b5875 sp 00007ffdf12be670 error 4 in ld-2.28.so[7fc990494000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[181109]: segfault at 8 ip 00007ffbfeacf875 sp 00007ffe68f1caa0 error 4 in ld-2.28.so[7ffbfeaae000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[181960]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[181960]: Failed to get EXE, ignoring: No such process
18[191475]: segfault at 8 ip 00007ff58a099875 sp 00007ffe8d2291b0 error 4 in ld-2.28.so[7ff58a078000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[192740]: segfault at 8 ip 00007f11aa557875 sp 00007ffcb62f25d0 error 4 in ld-2.28.so[7f11aa536000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[198434]: segfault at 0 ip 000055ba7371b9a1 sp 00007ffe6a7e1980 error 6 in 9[55ba73717000+7000]
Code: 44 24 08 31 c0 e8 3f cc ff ff 48 89 e6 48 89 df 49 89 c4 c7 00 00 00 00 00 41 ff d6 48 8b 0c 24 48 39 d9 74 74 48 85 ed 74 4f <66> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7[199284]: segfault at 563d48aeb000 ip 0000563d48aeb000 sp 00007ffc71eb31f8 error 14 in 7[563d48ceb000+1000]
Code: Unable to access opcode bytes at RIP 0x563d48aeafd6.
6[199804]: segfault at 61 ip 000055c98a46c070 sp 00007fffc7f5ae68 error 6 in 6[55c98a467000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 73 61 67 65 3a 20 25 73 20 4e 55 4d 42 45 52 5b 53 55 46 46 49
6[202513]: segfault at 8 ip 00007fa26c30d875 sp 00007fff845588b0 error 4 in ld-2.28.so[7fa26c2ec000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[202234]: segfault at 0 ip 0000560df4c12b47 sp 00007ffe32a6d660 error 6 in 9[560df4c0e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 203837:0:(mdc_request.c:1507:mdc_read_page()) lustre-MDT0002-mdc-ffffa0bf57cc7000: [0x280000403:0x1:0x0] lock enqueue fails: rc = -4
LustreError: 203837:0:(statahead.c:1814:is_first_dirent()) lustre: reading dir [0x280000403:0x1:0x0] at 0 stat_pid = 205359 : rc = -4
LustreError: 203837:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x280000403:0x127e:0x0] error: rc = -4
19[208369]: segfault at 8 ip 00007f75838f2875 sp 00007ffec73369d0 error 4 in ld-2.28.so[7f75838d1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[209138]: segfault at 1 ip 0000563b30aeb1b2 sp 00007ffe954d5638 error 4 in 5[563b30ae7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 24 10 48 89 04 24 <48> 8b 1b 4c 89 44 24 08 31 ff e8 7f d4 ff ff 4c 8b 54 24 18 41 52
LustreError: 212127:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0bf5911e000: inode [0x280000403:0x14c2:0x0] mdc close failed: rc = -13
10[215022]: segfault at 8 ip 00007f7ea347d875 sp 00007ffee16b61b0 error 4 in ld-2.28.so[7f7ea345c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 221061:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0bf57cc7000: inode [0x200000403:0x1dba:0x0] mdc close failed: rc = -13
1[225357]: segfault at 8 ip 00007f206c213875 sp 00007fff503f89a0 error 4 in ld-2.28.so[7f206c1f2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[226248]: segfault at e ip 00005637c3651a78 sp 00007ffe53bed4d8 error 6 in 7[5637c364b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 63 0e 10 14 00 00 00 b8 09 00 00 6c dc ff ff 86 00 00 00 00 02
7[228980]: segfault at 8 ip 00007fa5fb052875 sp 00007fff90419050 error 4 in ld-2.28.so[7fa5fb031000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[229836]: segfault at 8 ip 00007f52806ce875 sp 00007ffc49a4df20 error 4 in ld-2.28.so[7f52806ad000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6775:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 6775:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Link to test (2026-08-25 14:20)
racer test 1: racer on clients: centos-70.localnet DURATION=2700
LustreError: 1617283:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 1617283:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 0 PID: 1617283 Comm: ll_sa_1617259 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 1378301:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa064cc04a300 x1874492690848896/t4294967952(0) o101->7bfa0f71-c64b-4b40-8a8b-3241ffd216c7@0@lo:573/0 lens 376/816 e 0 to 0 dl 1787655598 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 1379290:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0640c123000: inode [0x200000402:0xb:0x0] mdc close failed: rc = -13
Lustre: 1376389:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1376389:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1376389:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1376389:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1376389:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1376389:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1376389:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1376389:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1376389:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1376389:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376389:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1376389:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376389:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 1376389:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376389:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1376389:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376389:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1376389:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 1384556:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0647914d000: inode [0x200000401:0x1e4:0x0] mdc close failed: rc = -13
Lustre: 1376390:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1376390:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 1376390:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1376390:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376390:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1376390:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376390:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1376390:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376390:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1376390:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1376390:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1376390:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 1381548:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1381548:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1381548:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 1381548:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 1379147:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1379147:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1379147:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1379147:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1379147:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1379147:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1379147:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1379147:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1379147:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1379147:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1379147:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 1379147:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1379147:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1379147:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1379147:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 1376391:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 585, rollback = 7
Lustre: 1376391:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 1376391:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1376391:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1376391:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1376391:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1376391:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/585/0, punch: 0/0/0, quota 1/3/0
Lustre: 1376391:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1376391:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1376391:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 1376391:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1376391:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
6[1407178]: segfault at 8 ip 00007fd33327c875 sp 00007ffcdeb2c080 error 4 in ld-2.28.so[7fd33325b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[1408604]: segfault at 5611b1643000 ip 00005611b1643000 sp 00007ffe8486be68 error 14 in 1[5611b1843000+1000]
Code: Unable to access opcode bytes at RIP 0x5611b1642fd6.
LustreError: 1408604:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0640c123000: inode [0x200000401:0x978:0x0] mdc close failed: rc = -13
Lustre: 1378367:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1378367:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 1378367:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1378367:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1378367:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1378367:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1378367:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1378367:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1378367:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1378367:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 1378367:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1378367:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 17 previous similar messages
19[1409990]: segfault at 0 ip 000055f241882b47 sp 00007ffcd0d28250 error 6 in 19[55f24187e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
14[1410769]: segfault at 8 ip 00007fd9c50ce875 sp 00007fff7e028cc0 error 4 in ld-2.28.so[7fd9c50ad000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[1418100]: segfault at 0 ip 0000555affc4eb47 sp 00007fff4fc71e60 error 6 in 4[555affc4a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1378837:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
traps: 9[1419386] trap invalid opcode ip:55f3f7083eea sp:7ffdf719cf38 error:0 in 9[55f3f707e000+7000]
Lustre: 1388636:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 1388636:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1388636:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1388636:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1388636:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1388636:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 40 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1388636:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 40 previous similar messages
12[1425072]: segfault at 8 ip 00007f4ae68d9875 sp 00007ffeff6a8260 error 4 in ld-2.28.so[7f4ae68b8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[1431610]: segfault at 0 ip 000055e4ff8c1b47 sp 00007ffe9296c790 error 6 in 6[55e4ff8bd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[1434561]: segfault at 8 ip 00007f8a64488875 sp 00007ffcf1baf3d0 error 4 in ld-2.28.so[7f8a64467000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[1434738]: segfault at 0 ip 000055d5dd4a1b47 sp 00007fff7f503a60 error 6 in 8[55d5dd49d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
11[1437125]: segfault at 0 ip 00005603001d8d50 sp 00007ffe2df437f8 error 6 in 17[5603001d4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1381548:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 1381548:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 87 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 87 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1381548:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 87 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 1381548:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 87 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 87 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 87 previous similar messages
7[1453870]: segfault at 8 ip 00007f3f35e44875 sp 00007ffe9ea032d0 error 4 in ld-2.28.so[7f3f35e23000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[1457883]: segfault at 8 ip 00007f1201acc875 sp 00007ffc3753fe80 error 4 in ld-2.28.so[7f1201aab000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[1465489]: segfault at 0 ip 000055f309320b47 sp 00007fffcaf0e960 error 6 in 18[55f30931c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[1486515]: segfault at 8 ip 00007f98034e6875 sp 00007fff6839a700 error 4 in ld-2.28.so[7f98034c5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1496339:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0647914d000: inode [0x200000401:0x2572:0x0] mdc close failed: rc = -13
4[1501940]: segfault at 8 ip 00007f9b5002f875 sp 00007ffce2956280 error 4 in ld-2.28.so[7f9b5000e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1381548:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 561, rollback = 7
Lustre: 1381548:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 141 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 141 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 1381548:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 141 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/561/0, punch: 0/0/0, quota 4/150/0
Lustre: 1381548:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 141 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 141 previous similar messages
Lustre: 1381548:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1381548:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 141 previous similar messages
Lustre: 1376128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
2[1510647]: segfault at 0 ip 000055b6b58f441c sp 00007ffe444a4138 error 6 in 2[55b6b58ef000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 70 65 2d 61 6c 77 61 79 73 <00> 63 00 63 2d 6d 61 79 62 65 00 63 6c 6f 63 61 6c 65 00 00 00 00
9[1520509]: segfault at 0 ip 000056143b6d4b47 sp 00007ffd11e4efc0 error 6 in 9[56143b6d0000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9[1520543]: segfault at 0 ip 00005572df8c3b47 sp 00007fffe156a5a0 error 6 in 9[5572df8bf000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 1376128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 1376128:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
0[1538815]: segfault at 8 ip 00007f87d2d16875 sp 00007fff6270d390 error 4 in ld-2.28.so[7f87d2cf5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[1546837]: segfault at 8 ip 00007fac64355875 sp 00007ffc7de65550 error 4 in ld-2.28.so[7fac64334000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[1547793]: segfault at 8 ip 00007fe660c48875 sp 00007ffc156086a0 error 4 in ld-2.28.so[7fe660c27000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1376128:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 1376128:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
18[1552471]: segfault at 8 ip 00007fd312b14875 sp 00007fff0aac9ff0 error 4 in ld-2.28.so[7fd312af3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[1552702]: segfault at 0 ip 000055b02e1abb47 sp 00007ffcd62ec050 error 6 in 4[55b02e1a7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3[1560167]: segfault at 0 ip 0000561646a05b47 sp 00007fff1c4e6300 error 6 in 17[561646a01000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
19[1566371]: segfault at 8 ip 00007fe6331e3875 sp 00007ffda2b444b0 error 4 in ld-2.28.so[7fe6331c2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 1568629:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0640c123000: inode [0x200000401:0x3d50:0x0] mdc close failed: rc = -13
Lustre: mdt_io00_003: service thread pid 1378837 was inactive for 41.920 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt_io00_001 state:I
task:ll_ost_out00_00 state:I stack:0 pid:1496598 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x20/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
Lustre: Skipped 2 previous similar messages
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
stack:0 pid:1376129 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
task:mdt_io00_003 state:I stack:0 pid:1378837 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? mdt_object_put+0x130/0x130 [mdt]
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_rename_source_lock+0x70/0x180 [mdt]
mdt_reint_rename+0xd17/0x34a0 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_check_lock+0x24b/0x4c0 [mdt]
mdt_reint_rename+0x1707/0x34a0 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_out00_00: service thread pid 1376395 was inactive for 42.032 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 1376105:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffffa064092a4200/0xe3c81fcda40de4cd lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x40ed:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xe3c81fcda40de34c expref: 1235 pid: 1496598 timeout: 26895 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt_io00_003: service thread pid 1378837 completed after 99.271s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffffa0647914d000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 1376101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787656633 with bad export cookie 16413403810023659712
Lustre: lustre-MDT0000-mdc-ffffa0647914d000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0000-mdc-ffffa0647914d000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt_io00_001: service thread pid 1376129 completed after 99.350s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: ll_ost_out00_00: service thread pid 1496598 completed after 99.353s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1376395:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000004b783238 ns: mdt-lustre-MDT0000_UUID lock: ffffa0645efaac00/0xe3c81fcda40df06c lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x4073:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0xe3c81fcda40de5d0 expref: 4 pid: 1376395 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: ll_ost_out00_00: service thread pid 1376395 completed after 99.353s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 1580415:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa0647914d000: inode [0x200000401:0x1:0x0] mdc close failed: rc = -108
LustreError: 1580006:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
LustreError: 1580006:0:(file.c:6167:ll_inode_revalidate_fini()) Skipped 7 previous similar messages
Lustre: lustre-MDT0000-mdc-ffffa0647914d000: Connection restored to 0@lo (at 0@lo)
12[1582963]: segfault at 8 ip 00007ff70f82b875 sp 00007ffe4f11fa60 error 4 in ld-2.28.so[7ff70f80a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[1594632]: segfault at 8 ip 00007f16a1d70875 sp 00007ffca42275a0 error 4 in ld-2.28.so[7f16a1d4f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[1594771]: segfault at 8 ip 00007fc47db2f875 sp 00007fffc5ff3c20 error 4 in ld-2.28.so[7fc47db0e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 1388636:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 567, rollback = 7
Lustre: 1388636:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 319 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 1388636:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 325 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 1388636:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 325 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/567/0, punch: 0/0/0, quota 4/150/0
Lustre: 1388636:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 325 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 1388636:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 325 previous similar messages
Lustre: 1388636:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 1388636:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 325 previous similar messages
8[1602099]: segfault at 0 ip 000055dbae70fb47 sp 00007ffccb8b1480 error 6 in 8[55dbae70b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[1607149]: segfault at 8 ip 00007f9147c2a875 sp 00007ffc14145f90 error 4 in ld-2.28.so[7f9147c09000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[1613950]: segfault at 8 ip 00007f735a0cd875 sp 00007ffef4fea030 error 4 in ld-2.28.so[7f735a0ac000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Link to test (2026-08-25 11:20)
racer test 1: racer on clients: centos-120.localnet DURATION=2700
LustreError: 519636:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 519636:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 11 PID: 519636 Comm: ll_sa_518515 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 4166602:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166602:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166602:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166602:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 4166602:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166602:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4168846:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9702150c1880 x1874410784250880/t4294971126(0) o101->45146981-af0f-417b-8aad-c5f1f4d98797@0@lo:239/0 lens 376/864 e 0 to 0 dl 1787577499 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 4168651:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4168651:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4168651:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 4168651:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
LustreError: 4174608:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9701f20db800: inode [0x200000401:0x197:0x0] mdc close failed: rc = -13
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166602:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166602:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 4166602:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166602:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166602:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166602:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166602:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4166602:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166602:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166602:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166602:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166602:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166604:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166604:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 4166604:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166604:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166604:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 4166604:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166604:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166604:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4166604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4168651:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 4168651:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 4168651:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4168651:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 4168651:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 4168651:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 4168651:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 4168651:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
4[4180238]: segfault at 0 ip 000055999e3f8b47 sp 00007ffda0ef49c0 error 6 in 4[55999e3f4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 4168651:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4168651:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4168651:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 4168651:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168651:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4168651:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 4168863:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4168863:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4168863:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4168863:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4168863:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4168863:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4168863:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages
14[4193763]: segfault at 8 ip 00007ff400c4e875 sp 00007ffc83af9c90 error 4 in ld-2.28.so[7ff400c2d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 2886:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000401:0x98f:0x0] mdc close failed: rc = -13
Lustre: 4168863:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 579, rollback = 7
Lustre: 4168863:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 21 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4168863:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4168863:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/579/0, punch: 0/0/0, quota 1/3/0
Lustre: 4168863:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4168863:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 21 previous similar messages
Lustre: 4168863:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4168863:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 21 previous similar messages
10[19601]: segfault at 8 ip 00007f24999d4875 sp 00007ffe6b2664d0 error 4 in ld-2.28.so[7f24999b3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[19930]: segfault at 8 ip 00007fbed8a25875 sp 00007fff5167caa0 error 4 in ld-2.28.so[7fbed8a04000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[20059]: segfault at 8 ip 00007f50ce826875 sp 00007ffe472408b0 error 4 in ld-2.28.so[7f50ce805000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 21371:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000401:0xe3c:0x0] mdc close failed: rc = -13
19[24074]: segfault at 8 ip 00007f6aa3609875 sp 00007fff9c6ebb00 error 4 in ld-2.28.so[7f6aa35e8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: lustre-OST0000: already connected client lustre-MDT0000-mdtlov_UUID (at 0@lo) with handle 0x6864330f9ff1580b. Rejecting client with the same UUID trying to reconnect with handle 0xbd5080b397e864e9
16[24810]: segfault at 8 ip 00007ff657b2f875 sp 00007ffcd1688010 error 4 in ld-2.28.so[7ff657b0e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 71 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 71 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 71 previous similar messages
6[38189]: segfault at a2614ec0 ip 00005644d91f1616 sp 00007ffea2614ec0 error 6 in 6[5644d91ef000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[38124]: segfault at 8 ip 00007fc5d9b8f875 sp 00007fff7a02aa00 error 4 in ld-2.28.so[7fc5d9b6e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[39118]: segfault at 8 ip 00007f4ac0a24875 sp 00007ffcfb530590 error 4 in ld-2.28.so[7f4ac0a03000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166342:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
18[79789]: segfault at 8 ip 00007f3db01ad875 sp 00007ffd7789b380 error 4 in ld-2.28.so[7f3db018c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 21320:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 21320:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 117 previous similar messages
Lustre: 21320:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 21320:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 121 previous similar messages
Lustre: 21320:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 21320:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 121 previous similar messages
Lustre: 21320:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 21320:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 121 previous similar messages
Lustre: 21320:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 21320:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 121 previous similar messages
Lustre: 21320:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 21320:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 121 previous similar messages
1[98939]: segfault at 0 ip 000055ea4036ca88 sp 00007ffc13df6d20 error 6 in 1[55ea4036b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[102614]: segfault at 0 ip 0000564d047f8b47 sp 00007fff39734c00 error 6 in 16[564d047f4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[102889]: segfault at 8 ip 00007f4b0e967875 sp 00007fff12bcb030 error 4 in ld-2.28.so[7f4b0e946000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[103954]: segfault at 55f8567cc000 ip 000055f8567cc000 sp 00007fff33bb6368 error 14 in 4[55f8569cc000+1000]
Code: Unable to access opcode bytes at RIP 0x55f8567cbfd6.
18[106004]: segfault at 0 ip 000056463a07ab47 sp 00007ffc950e8d90 error 6 in 18[56463a076000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[115847]: segfault at 0 ip 0000558d9ec2c200 sp 00007ffd136199a8 error 6 in 16[558d9ec2a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[127163]: segfault at 558235322000 ip 0000558235322000 sp 00007ffda49b3898 error 14 in 8[558235522000+1000]
Code: Unable to access opcode bytes at RIP 0x558235321fd6.
10[134694]: segfault at 8 ip 00007ff1217d9875 sp 00007ffed91ee820 error 4 in ld-2.28.so[7ff1217b8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[135669]: segfault at 8 ip 00007f0a76513875 sp 00007ffd3a5d1d20 error 4 in ld-2.28.so[7f0a764f2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4170181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 4170181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
0[139542]: segfault at 8 ip 00007fc3196ec875 sp 00007ffdf7280240 error 4 in ld-2.28.so[7fc3196cb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: mdt00_011: service thread pid 4168867 was inactive for 41.089 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt00_015 state:I
task:mdt_out00_004 state:I stack:0 pid:31940 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
Lustre: mdt_out00_002: service thread pid 4168655 was inactive for 41.094 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x20/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: Skipped 2 previous similar messages
stack:0 pid:4171230 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x41b/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_object_find+0x21/0x30 [obdclass]
mdt_reint_unlink+0x234/0x1a30 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
task:mdt00_011 state:I
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
stack:0 pid:4168867 ppid:2 flags:0x80004080
kthread+0x1de/0x210
Call Trace:
? set_kthread_struct+0x70/0x70
__schedule+0x351/0xcb0
ret_from_fork+0x1f/0x30
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
? mdt_object_find+0x10b/0x480 [mdt]
? ldlm_request_cancel+0xb8/0xba0 [ptlrpc]
mdt_object_find_lock+0x7a/0x1d0 [mdt]
mdt_reint_setxattr+0x1bb/0x1880 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: mdt00_007: service thread pid 4168642 was inactive for 44.032 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 9 previous similar messages
Lustre: lustre-OST0003-osc-ffff97020c103000: disconnect after 22s idle
Lustre: lustre-OST0000-osc-ffff9701f20db800: disconnect after 22s idle
Lustre: Skipped 1 previous similar message
LustreError: 4166319:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff970243daf800/0x6864330fa068f3cb lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x37a0:0x0].0x0 bits 0x13/0x0 rrc: 6 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x6864330fa068f362 expref: 1204 pid: 4166335 timeout: 38083 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_011: service thread pid 4168867 completed after 102.539s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 4166343:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff970225354280 x1874410898458752/t0(0) o104->lustre-MDT0000@0@lo:15/16 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
LustreError: 4170591:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 00000000421904f3 ns: mdt-lustre-MDT0000_UUID lock: ffff9701fa2ba400/0x6864330fa068fe28 lrc: 4/0,0 mode: PR/PR res: [0x200000401:0x1:0x0].0x0 bits 0x13/0x0 rrc: 21 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x6864330fa068fe0c expref: 603 pid: 4170591 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_014: service thread pid 4170591 completed after 101.407s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9701f20db800: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: mdt00_007: service thread pid 4168642 completed after 101.412s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff9701f20db800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_018: service thread pid 4173492 completed after 101.412s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_004: service thread pid 4168473 completed after 101.409s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_003: service thread pid 4167151 completed after 101.431s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: Skipped 2 previous similar messages
Lustre: mdt00_002: service thread pid 4166331 completed after 101.432s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9701f20db800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
Lustre: mdt_out00_001: service thread pid 4166335 completed after 102.607s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_010: service thread pid 4168862 completed after 102.603s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_004: service thread pid 31940 completed after 102.559s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_012: service thread pid 4169497 completed after 102.600s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_out00_002: service thread pid 4168655 completed after 102.584s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 143471:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000401:0x1:0x0] error: rc = -5
Lustre: mdt_io00_001: service thread pid 4166343 completed after 102.614s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_015: service thread pid 4171230 completed after 102.560s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 143501:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9701f20db800: inode [0x200000402:0x347f:0x0] mdc close failed: rc = -108
Lustre: lustre-MDT0000-mdc-ffff9701f20db800: Connection restored to 0@lo (at 0@lo)
12[156358]: segfault at 55e5b3228000 ip 000055e5b3228000 sp 00007ffdb5d5a5c8 error 14 in 12[55e5b3428000+1000]
Code: Unable to access opcode bytes at RIP 0x55e5b3227fd6.
10[161588]: segfault at 8 ip 00007f6d27b2a875 sp 00007ffd722dafa0 error 4 in ld-2.28.so[7f6d27b09000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 163229:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000403:0x604:0x0] mdc close failed: rc = -13
LustreError: 163229:0:(file.c:251:ll_close_inode_openhandle()) Skipped 9 previous similar messages
1[166133]: segfault at 0 ip 0000555ba12e6b47 sp 00007ffced8937c0 error 6 in 1[555ba12e2000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 173278:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000403:0x9bb:0x0] mdc close failed: rc = -13
Lustre: 4166344:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
14[188958]: segfault at 8 ip 00007fa09e9ec875 sp 00007ffda1eba420 error 4 in ld-2.28.so[7fa09e9cb000+2f000]
14[188894]: segfault at 8 ip 00007f9b68d57875 sp 00007ffeb80ca9a0 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7f9b68d36000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[189028]: segfault at 8 ip 00007f0dfacf7875 sp 00007ffe41cdfc70 error 4 in ld-2.28.so[7f0dfacd6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[189123]: segfault at 8 ip 00007f7b037cf875 sp 00007ffe61344510 error 4 in ld-2.28.so[7f7b037ae000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 189123:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000403:0xeaf:0x0] mdc close failed: rc = -13
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 303 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 309 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 309 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 309 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 309 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 309 previous similar messages
19[194534]: segfault at 8 ip 00007f6681e3f875 sp 00007ffe36a4cbf0 error 4 in ld-2.28.so[7f6681e1e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[200384]: segfault at 8 ip 00007f136cb29875 sp 00007fffa72a9c10 error 4 in ld-2.28.so[7f136cb08000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[204407]: segfault at 0 ip 000055e642ac6b47 sp 00007ffec8f97750 error 6 in 14[55e642ac2000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[205963]: segfault at 8 ip 00007fd7d25f2875 sp 00007ffdcb2d9030 error 4 in ld-2.28.so[7fd7d25d1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 4 previous similar messages
5[215695]: segfault at 0 ip 0000564062c77b47 sp 00007ffe2342c010 error 6 in 5[564062c73000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[216235]: segfault at 8 ip 00007fa4b845c875 sp 00007fff58c76f90 error 4 in ld-2.28.so[7fa4b843b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[216711]: segfault at 8 ip 00007f57ed1c1875 sp 00007ffe1279e560 error 4 in ld-2.28.so[7f57ed1a0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[226685]: segfault at 8 ip 00007f098d97a875 sp 00007ffcc4269530 error 4 in ld-2.28.so[7f098d959000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[249298]: segfault at 0 ip 000055579baeeb47 sp 00007ffc95b99480 error 6 in 1[55579baea000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[249597]: segfault at 8 ip 00007fd1e6129875 sp 00007ffcd4bc48f0 error 4 in ld-2.28.so[7fd1e6108000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[249698]: segfault at 8 ip 00007f907c96d875 sp 00007ffdeb1634f0 error 4 in ld-2.28.so[7f907c94c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[251850]: segfault at 8 ip 00007f897138b875 sp 00007fffe9118f20 error 4 in ld-2.28.so[7f897136a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[253919]: segfault at 0 ip 000055743b538b47 sp 00007ffff8f6a0e0 error 6 in 18[55743b534000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
11[265155]: segfault at 0 ip 000055baec0f8b47 sp 00007ffe26e17df0 error 6 in 11[55baec0f4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[265347]: segfault at 8 ip 00007f1450c48875 sp 00007fff88109540 error 4 in ld-2.28.so[7f1450c27000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[272667]: segfault at 8 ip 00007f5c54450875 sp 00007ffd227416a0 error 4 in ld-2.28.so[7f5c5442f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[275044]: segfault at 8 ip 00007f3bf012a875 sp 00007ffd1e683690 error 4 in ld-2.28.so[7f3bf0109000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[282851]: segfault at 0 ip 00005592b88efb47 sp 00007ffe7a7af350 error 6 in 19[5592b88eb000+7000]
19[282890]: segfault at 0 ip 000055980dee4b47 sp 00007ffd7e281480 error 6
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
in 19[55980dee0000+7000]
Code: Unable to access opcode bytes at RIP 0x55980dee4b1d.
LustreError: 282493:0:(statahead.c:2416:start_statahead_thread()) lustre: invalid pattern 0X0.
3[284479]: segfault at 8 ip 00007f2201745875 sp 00007ffef78001f0 error 4 in ld-2.28.so[7f2201724000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[287113]: segfault at 55b85524a000 ip 000055b85524a000 sp 00007ffc5c2f8f18 error 14 in 13[55b85544a000+1000]
Code: Unable to access opcode bytes at RIP 0x55b855249fd6.
LustreError: 289611:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000401:0x68e5:0x0] mdc close failed: rc = -13
LustreError: 293255:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9701f20db800: inode [0x200000403:0x31b3:0x0] mdc close failed: rc = -13
18[301791]: segfault at 0 ip 0000559e46452b47 sp 00007ffeba4fabd0 error 6 in 18 (deleted)[559e4644e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 4166342:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 4166342:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
7[324438]: segfault at 8 ip 00007f1622917875 sp 00007ffc758593f0 error 4 in ld-2.28.so[7f16228f6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[325165]: segfault at 0 ip 0000563c7cf74b47 sp 00007fff7649f150 error 6 in 16[563c7cf70000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[325544]: segfault at 8 ip 00007fb6a8d82875 sp 00007ffc05879cc0 error 4 in ld-2.28.so[7fb6a8d61000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[325832]: segfault at 8 ip 00007fd9eebc4875 sp 00007ffea6854bd0 error 4 in ld-2.28.so[7fd9eeba3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[327070]: segfault at 8 ip 00007f0f1873b875 sp 00007fff507b29b0 error 4 in ld-2.28.so[7f0f1871a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[327659]: segfault at 56133f92d000 ip 000056133f92d000 sp 00007ffd5c02d4b8 error 14 in 11[56133fb2d000+1000]
Code: Unable to access opcode bytes at RIP 0x56133f92cfd6.
13[333479]: segfault at 0 ip 000055e6c29adb47 sp 00007fff76307af0 error 6 in 13[55e6c29a9000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[333459]: segfault at 0 ip 000055f09e47bb47 sp 00007ffef1eb5a60 error 6 in 13[55f09e477000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[333756]: segfault at 8 ip 00007ff50d8c6875 sp 00007ffee8580a90 error 4 in ld-2.28.so[7ff50d8a5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[334124]: segfault at 8 ip 00007fa8d4cf3875 sp 00007ffeed7b8180 error 4 in ld-2.28.so[7fa8d4cd2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166603:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 441 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 445 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166603:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 445 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4166603:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 445 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 445 previous similar messages
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166603:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 445 previous similar messages
7[339356]: segfault at 8 ip 00007f70a960a875 sp 00007ffe66b3f3c0 error 4 in ld-2.28.so[7f70a95e9000+2f000]
7[339425]: segfault at 8 ip 00007f31ee739875 sp 00007ffc35413af0 error 4 in ld-2.28.so[7f31ee718000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[354451]: segfault at 0 ip 0000000000000000 sp 00007fff9b8e96d8 error 14 in 6[559f847c6000+7000]
Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
11[360306]: segfault at 8 ip 00007f7957472875 sp 00007ffcb6f33e50 error 4 in ld-2.28.so[7f7957451000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[370226]: segfault at 8 ip 00007fde73ca2875 sp 00007ffce65ad5f0 error 4 in ld-2.28.so[7fde73c81000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[372407]: segfault at 0 ip 00005644625bab47 sp 00007ffef1a766c0 error 6 in 19[5644625b6000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[379500]: segfault at 0 ip 0000557d78604b47 sp 00007ffcdeb0d7e0 error 6 in 6[557d78600000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[383048]: segfault at 0 ip 0000559ad375eb47 sp 00007ffd16f6dd90 error 6 in 10[559ad375a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7[384423]: segfault at 0 ip 000055c43b904b47 sp 00007ffd5fac1e30 error 6 in 7[55c43b900000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[385551]: segfault at 561be13ec000 ip 0000561be13ec000 sp 00007fff7568fe18 error 14 in 10[561be15ec000+1000]
Code: Unable to access opcode bytes at RIP 0x561be13ebfd6.
traps: 11[403920] general protection fault ip:55daa50a1a34 sp:7ffdd7564b28 error:0 in 11[55daa509b000+7000]
LustreError: 403920:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff97020c103000: inode [0x200000403:0x56d1:0x0] mdc close failed: rc = -13
11[407500]: segfault at 8 ip 00007fea95415875 sp 00007ffccbea6b60 error 4 in ld-2.28.so[7fea953f4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4170181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 4170181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
4[410003]: segfault at 8 ip 00007f0526627875 sp 00007fff5303aad0 error 4 in ld-2.28.so[7f0526606000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[416370]: segfault at 8 ip 00007f14c802f875 sp 00007ffe369e58c0 error 4 in ld-2.28.so[7f14c800e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 420418:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9701f20db800: inode [0x200000401:0x9608:0x0] mdc close failed: rc = -13
3[437525]: segfault at 0 ip 000055703980a35c sp 00007fffbfdc8e98 error 6 in 3[557039805000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 18 f5 40 38 2e 33 30 <00> 43 00 77 72 69 74 65 20 65 72 72 6f 72 00 25 73 3a 20 25 73 00
traps: 4[439352] trap invalid opcode ip:55fbb559f52e sp:7ffeef3ecd08 error:0 in 4[55fbb559a000+7000]
11[441181]: segfault at 0 ip 000055a8f7e10b47 sp 00007ffe131143a0 error 6 in 11[55a8f7e0c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[446468]: segfault at 0 ip 00005583e5a0bb47 sp 00007fffb86d8050 error 6 in 8[5583e5a07000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18[453531]: segfault at 8 ip 00007f01f340d875 sp 00007fff673a99a0 error 4 in ld-2.28.so[7f01f33ec000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[456180]: segfault at 8 ip 00007f3716fb5875 sp 00007ffc2a1f9eb0 error 4 in ld-2.28.so[7f3716f94000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[461078]: segfault at 0 ip 000055d86037a0b0 sp 00007ffcc0e0e688 error 6 in 9[55d860378000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[462100]: segfault at 8 ip 00007f94c2dc3875 sp 00007fff11b38620 error 4 in ld-2.28.so[7f94c2da2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
4[470257]: segfault at 8 ip 00007f803defe875 sp 00007fffaee6f880 error 4 in ld-2.28.so[7f803dedd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[478760]: segfault at 0 ip 0000565191301b47 sp 00007ffc6eccdcb0 error 6 in 8[5651912fd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3[483227]: segfault at 0 ip 000055ed1b678b47 sp 00007ffc07d107e0 error 6 in 3[55ed1b674000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1[486800]: segfault at 8 ip 00007f240a048875 sp 00007fffbd5fdab0 error 4 in ld-2.28.so[7f240a027000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[486917]: segfault at 8 ip 00007f56ace6e875 sp 00007ffd4029d790 error 4 in ld-2.28.so[7f56ace4d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166604:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 4166604:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 347 previous similar messages
Lustre: 4166604:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 4166604:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 350 previous similar messages
Lustre: 4166604:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 4166604:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 350 previous similar messages
Lustre: 4166604:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 4166604:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 350 previous similar messages
Lustre: 4166604:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 4166604:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 350 previous similar messages
Lustre: 4166604:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 4166604:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 350 previous similar messages
6[489913]: segfault at 0 ip 000055f741c81b47 sp 00007fff5f293990 error 6 in 4[55f741c7d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[490332]: segfault at 8 ip 00007f1363f34875 sp 00007ffcf5282bb0 error 4 in ld-2.28.so[7f1363f13000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[495840]: segfault at 8 ip 00007f516c3e9875 sp 00007ffd081f9120 error 4 in ld-2.28.so[7f516c3c8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[498964]: segfault at 8 ip 00007f45824f1875 sp 00007ffdd651c230 error 4 in ld-2.28.so[7f45824d0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 4166343:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
12[515727]: segfault at 8 ip 00007fc1d1379875 sp 00007ffce2289610 error 4 in ld-2.28.so[7fc1d1358000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Link to test (2026-08-24 13:58)
racer test 1: racer on clients: centos-25.localnet DURATION=2700
LustreError: 3685979:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 3685979:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 11 PID: 3685979 Comm: ll_sa_3685925 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 3094004:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3094004:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3094004:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3094004:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3094003:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 3094003:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3094003:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 3094003:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3094003:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3094003:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3094003:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3097377:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffffa0063c26b100 x1874225092868608/t4294970667(0) o101->9d111a4e-0cf1-4533-9199-9d1ec1a4891c@0@lo:561/0 lens 384/864 e 0 to 0 dl 1787400396 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 3096343:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 3096343:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096343:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 17/17/1, xattr_set: 26/520/0
Lustre: 3096343:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096343:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 1/3/0
Lustre: 3096343:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096343:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 7/117/0, delete: 3/6/0
Lustre: 3096343:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096343:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 3096343:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3097608:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3097608:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 3097608:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3097608:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3097608:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3097608:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3097608:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3097608:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3097608:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3097608:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3097608:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3097608:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 3097608:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 3097608:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3097608:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3097608:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 3097608:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 3097608:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 3097608:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/555/0, punch: 0/0/0, quota 4/150/0
Lustre: 3097608:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 3097608:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3097608:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 3097608:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3097608:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 3096130:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3096130:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 3096163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 537, rollback = 7
Lustre: 3096163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3096163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3096163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 3096163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/537/0, punch: 0/0/0, quota 4/150/0
Lustre: 3096163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3096163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3096163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 3094004:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3094004:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3094004:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3094004:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
9[3111687]: segfault at 0 ip 0000557a1e642b47 sp 00007ffe8e2a8c10 error 6 in 9[557a1e63e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
11[3112072]: segfault at 8 ip 00007f6cb354a875 sp 00007ffcf610e120 error 4 in ld-2.28.so[7f6cb3529000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 3112072:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00728970000: inode [0x200000401:0x49b:0x0] mdc close failed: rc = -13
Lustre: 3097737:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3097737:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 25 previous similar messages
Lustre: 3097737:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3097737:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3097737:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3097737:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3097737:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3097737:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3097737:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3097737:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 25 previous similar messages
Lustre: 3097737:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3097737:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 25 previous similar messages
LustreError: 3124620:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0x9d8:0x0] mdc close failed: rc = -13
LustreError: 3127777:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0xc53:0x0] mdc close failed: rc = -13
Lustre: lustre-OST0000-osc-ffffa00728970000: disconnect after 24s idle
Lustre: 3093742:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 3093742:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 3094004:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 543, rollback = 7
Lustre: 3094004:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 15 previous similar messages
Lustre: 3094004:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 3094004:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3094004:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 3094004:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/543/0, punch: 0/0/0, quota 1/3/0
Lustre: 3094004:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 3094004:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 17 previous similar messages
Lustre: 3094004:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3094004:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 17 previous similar messages
3[3160123]: segfault at 8 ip 00007f9d9e29a875 sp 00007ffd2100bde0 error 4 in ld-2.28.so[7f9d9e279000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3096163:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3096163:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 33 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3096163:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3096163:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 3096163:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3096163:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 33 previous similar messages
Lustre: 3096163:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3096163:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 33 previous similar messages
17[3172340]: segfault at 8 ip 00007f0c9af02875 sp 00007ffe14849e20 error 4 in ld-2.28.so[7f0c9aee1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
INFO: task rm:3118821 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:rm state:D stack:0 pid:3118821 ppid:3095554 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
? __might_sleep+0x60/0xc0
down_write+0x84/0xd0
do_unlinkat+0x177/0x440
__x64_sys_unlinkat+0x50/0xa0
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f036ded213b
Code: Unable to access opcode bytes at RIP 0x7f036ded2111.
RSP: 002b:00007ffedb0f02d8 EFLAGS: 00000256 ORIG_RAX: 0000000000000107
RAX: ffffffffffffffda RBX: 00005571f670be40 RCX: 00007f036ded213b
RDX: 0000000000000000 RSI: 00005571f670bf48 RDI: 0000000000000004
RBP: 00005571f660a440 R08: 0000000000000003 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000256 R12: 00007ffedb0f04c0
R13: 0000000000000000 R14: 00005571f670be40 R15: 0000000000000000
INFO: task mrename:3118888 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mrename state:D stack:0 pid:3118888 ppid:3095573 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
down_write+0x84/0xd0
lock_rename+0x125/0x160
do_renameat2+0x326/0x750
__x64_sys_rename+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f87d00856cb
Code: Unable to access opcode bytes at RIP 0x7f87d00856a1.
RSP: 002b:00007ffd5d4fa0a8 EFLAGS: 00000202 ORIG_RAX: 0000000000000052
RAX: ffffffffffffffda RBX: 00007ffd5d4fa198 RCX: 00007f87d00856cb
RDX: 00007ffd5d4fa1b8 RSI: 00007ffd5d4fad68 RDI: 00007ffd5d4fad4f
RBP: 0000000000400800 R08: 00007f87d03d4d20 R09: 00007f87d03d4d20
R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000400710
R13: 00007ffd5d4fa190 R14: 0000000000000000 R15: 0000000000000000
INFO: task lfs:3118923 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:lfs state:D stack:0 pid:3118923 ppid:3095552 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
down_write+0x84/0xd0
vfs_setxattr+0x66/0x260
setxattr+0x15f/0x230
? slab_free_freelist_hook.isra.49+0x111/0x230
? kmem_cache_free+0x431/0x4c0
? putname+0x72/0xa0
? __audit_syscall_entry+0xda/0x200
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
__x64_sys_fsetxattr+0x11d/0x170
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f71171ba68e
Code: Unable to access opcode bytes at RIP 0x7f71171ba664.
RSP: 002b:00007ffe41faffe8 EFLAGS: 00000206 ORIG_RAX: 00000000000000be
RAX: ffffffffffffffda RBX: 00000000008fd570 RCX: 00007f71171ba68e
RDX: 00000000008fd570 RSI: 0000000000462b3e RDI: 0000000000000003
RBP: 0000000000000000 R08: 0000000000000000 R09: 00000000008fc02a
R10: 0000000000000160 R11: 0000000000000206 R12: 0000000000000003
R13: 00007ffe41fbed61 R14: 00000000000001b6 R15: 0000000000000000
INFO: task chmod:3118987 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:3118987 ppid:3095581 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
? putname+0x72/0xa0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
? __might_sleep+0x60/0xc0
down_write+0x84/0xd0
chmod_common+0x73/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f2d67ba7258
Code: Unable to access opcode bytes at RIP 0x7f2d67ba722e.
RSP: 002b:00007ffc70151178 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f2d67ba7258
RDX: 0000000000000dab RSI: 0000562f9f96b500 RDI: 00000000ffffff9c
RBP: 0000562f9f96b470 R08: 0000000000000000 R09: 0000000000000dab
R10: 0000000000000fff R11: 0000000000000246 R12: 0000562f9f0a7bb4
R13: 0000562f9f96b500 R14: 0000562f9f96c730 R15: 0000000000004000
INFO: task chown:3118991 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:3118991 ppid:3095616 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
? kmem_cache_free+0x3a1/0x4c0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
down_write+0x84/0xd0
chown_common.isra.13+0x11d/0x2c0
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f8709bd4fbe
Code: Unable to access opcode bytes at RIP 0x7f8709bd4f94.
RSP: 002b:00007ffda007c6b8 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffda007c8f0 RCX: 00007f8709bd4fbe
RDX: 0000000000007227 RSI: 00005599569f9d20 RDI: 00000000ffffff9c
RBP: 00005599569faf50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 00000000000033a9 R11: 0000000000000246 R12: 00000000ffffffff
R13: 00005599569fafc8 R14: 0000000000007227 R15: 00000000000033a9
INFO: task cat:3119013 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:cat state:D stack:0 pid:3119013 ppid:3095559 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? cfs_trace_unlock_tcd+0x2c/0xb0 [libcfs]
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
down_read+0xa9/0x2c0
lookup_slow+0x34/0x80
walk_component+0x264/0x4e0
link_path_walk+0x417/0x890
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0x135/0x500
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fdfb2d59332
Code: Unable to access opcode bytes at RIP 0x7fdfb2d59308.
RSP: 002b:00007ffd4e9dedc0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 00000000ffffffff RCX: 00007fdfb2d59332
RDX: 0000000000000000 RSI: 00007ffd4e9e0db5 RDI: 00000000ffffff9c
RBP: 0000000000000001 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 000055b8e540f790
R13: 00007ffd4e9df070 R14: 0000000000000000 R15: 0000000000400000
INFO: task setfattr:3119068 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:setfattr state:D stack:0 pid:3119068 ppid:3095567 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
down_write+0x84/0xd0
vfs_setxattr+0x66/0x260
setxattr+0x15f/0x230
? __check_object_size+0x10e/0x26d
? strncpy_from_user+0x61/0x2d0
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
path_setxattr+0x106/0x140
__x64_sys_setxattr+0x33/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f81d471280e
Code: Unable to access opcode bytes at RIP 0x7f81d47127e4.
RSP: 002b:00007fff8cb99f38 EFLAGS: 00000246 ORIG_RAX: 00000000000000bc
RAX: ffffffffffffffda RBX: 0000556398653450 RCX: 00007f81d471280e
RDX: 0000556398653450 RSI: 00007fff8cb9ad7a RDI: 00007fff8cb9adaf
RBP: 00007fff8cb9adaf R08: 0000000000000000 R09: 000055639865202e
R10: 000000000000002a R11: 0000000000000246 R12: 00007fff8cb9ad7a
R13: 00007fff8cb9a050 R14: 0000000000000000 R15: 0000000000000000
INFO: task setfattr:3119149 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:setfattr state:D stack:0 pid:3119149 ppid:3095565 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
? kmem_cache_free+0x3a1/0x4c0
down_write+0x84/0xd0
vfs_removexattr+0x4c/0x1b0
removexattr+0x5e/0xb0
? __check_object_size+0x10e/0x26d
? strncpy_from_user+0x61/0x2d0
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
? __sb_start_write+0x55/0x140
? mnt_want_write+0x3c/0x80
path_removexattr+0xed/0x120
__x64_sys_removexattr+0x23/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0c076767db
Code: Unable to access opcode bytes at RIP 0x7f0c076767b1.
RSP: 002b:00007ffc9d6c9b08 EFLAGS: 00000202 ORIG_RAX: 00000000000000c5
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f0c076767db
RDX: 0000000000000000 RSI: 00007ffc9d6cbda7 RDI: 00007ffc9d6cbdaf
RBP: 00007ffc9d6cbdaf R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000202 R12: 00007ffc9d6cbda7
R13: 00007ffc9d6c9c20 R14: 0000000000000000 R15: 0000000000000000
INFO: task setfattr:3119287 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:setfattr state:D stack:0 pid:3119287 ppid:3095651 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
down_write+0x84/0xd0
vfs_setxattr+0x66/0x260
setxattr+0x15f/0x230
? __check_object_size+0x10e/0x26d
? strncpy_from_user+0x61/0x2d0
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
path_setxattr+0x106/0x140
__x64_sys_setxattr+0x33/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ffa23c4180e
Code: Unable to access opcode bytes at RIP 0x7ffa23c417e4.
RSP: 002b:00007ffe160ab9a8 EFLAGS: 00000246 ORIG_RAX: 00000000000000bc
RAX: ffffffffffffffda RBX: 0000557ff966c450 RCX: 00007ffa23c4180e
RDX: 0000557ff966c450 RSI: 00007ffe160acd78 RDI: 00007ffe160acdaf
RBP: 00007ffe160acdaf R08: 0000000000000000 R09: 0000557ff966b02e
R10: 000000000000002b R11: 0000000000000246 R12: 00007ffe160acd78
R13: 00007ffe160abac0 R14: 0000000000000000 R15: 0000000000000000
INFO: task mkdir:3119339 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:mkdir state:D stack:0 pid:3119339 ppid:3095553 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_write_slowpath+0x5d7/0xa40
? __might_sleep+0x60/0xc0
down_write+0x84/0xd0
filename_create+0x96/0x220
do_mkdirat+0x77/0x160
__x64_sys_mkdir+0x23/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fae16559d4b
Code: Unable to access opcode bytes at RIP 0x7fae16559d21.
RSP: 002b:00007ffe5a204e98 EFLAGS: 00000246 ORIG_RAX: 0000000000000053
RAX: ffffffffffffffda RBX: 00007ffe5a206db2 RCX: 00007fae16559d4b
RDX: 0000000000000000 RSI: 00000000000001ff RDI: 00007ffe5a206dc7
RBP: 00007ffe5a206dc7 R08: 00000000000001ff R09: 000055c8a4343270
R10: 0000000000000000 R11: 0000000000000246 R12: 00000000000001ff
R13: 000055c8a4343290 R14: 00007ffe5a205010 R15: 00007ffe5a2051a8
Lustre: 3093743:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 3093743:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
9[3181909]: segfault at 249 ip 0000561e580263f6 sp 00007fff38830640 error 6 in 9[561e58022000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <43> 10 4c 8b 43 08 4c 8b 6b 30 4c 8b 63 28 4c 89 54 24 18 4c 8b 7b
8[3206190]: segfault at 8 ip 00007f03572b4875 sp 00007ffd69c2e280 error 4 in ld-2.28.so[7f0357293000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3093741:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3093741:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
5[3224377]: segfault at 8 ip 00007f4843c60875 sp 00007ffddab02ff0 error 4 in ld-2.28.so[7f4843c3f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[3232366]: segfault at 8 ip 00007f2bcac52875 sp 00007ffc56cbc570 error 4 in ld-2.28.so[7f2bcac31000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3094002:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3094002:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 109 previous similar messages
Lustre: 3094002:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3094002:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 115 previous similar messages
Lustre: 3094002:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 3094002:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 115 previous similar messages
Lustre: 3094002:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3094002:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 115 previous similar messages
Lustre: 3094002:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3094002:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 115 previous similar messages
Lustre: 3094002:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3094002:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 115 previous similar messages
13[3240500]: segfault at 8 ip 00007f5c43b57875 sp 00007ffea9e99be0 error 4 in ld-2.28.so[7f5c43b36000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 3244600:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0x4f6d:0x0] mdc close failed: rc = -13
LustreError: 3252811:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0x5431:0x0] mdc close failed: rc = -13
2[3255509]: segfault at 8 ip 00007f12a600c875 sp 00007ffc06dfaac0 error 4 in ld-2.28.so[7f12a5feb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[3255557]: segfault at 8 ip 00007f7d9f53d875 sp 00007ffe0969ba80 error 4 in ld-2.28.so[7f7d9f51c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[3258210]: segfault at 8 ip 00007f22f9850875 sp 00007fff8ad80e30 error 4 in ld-2.28.so[7f22f982f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[3269450]: segfault at 55acd6388f24 ip 000055acd6388f30 sp 00007ffe2ec31390 error 7 in 8[55acd6384000+7000]
Code: 2e 0f 1f 84 00 00 00 00 00 90 f3 0f 1e fa 48 8b 15 45 2c 20 00 31 f6 e9 ae c8 ff ff 00 00 f3 0f 1e fa 48 83 ec 08 48 83 c4 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 3093742:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3093742:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 3274690:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0x6076:0x0] mdc close failed: rc = -13
Lustre: lustre-OST0002-osc-ffffa00728970000: disconnect after 21s idle
Lustre: Skipped 1 previous similar message
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
19[3293004]: segfault at 8 ip 00007fa150342875 sp 00007fffec65c300 error 4 in ld-2.28.so[7fa150321000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[3304055]: segfault at 8 ip 00007f060001f875 sp 00007ffc45c7eca0 error 4 in ld-2.28.so[7f05ffffe000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[3329200]: segfault at 8 ip 00007f9c3a87a875 sp 00007ffc7571fff0 error 4 in ld-2.28.so[7f9c3a859000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[3335180]: segfault at 1b30 ip 0000000000001b30 sp 00007fff1dcbf5a0 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
14[3373644]: segfault at 8 ip 00007fa83da1b875 sp 00007ffe402a5d70 error 4 in ld-2.28.so[7fa83d9fa000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3093743:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3093743:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 4 previous similar messages
Lustre: 3093743:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 3093743:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 349 previous similar messages
Lustre: 3093743:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0
Lustre: 3093743:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 349 previous similar messages
Lustre: 3093743:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 7/249/0
Lustre: 3093743:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 349 previous similar messages
Lustre: 3093743:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 7/117/1, delete: 3/6/1
Lustre: 3093743:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 349 previous similar messages
Lustre: 3093743:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 3093743:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 349 previous similar messages
Lustre: 3094004:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 3094004:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 341 previous similar messages
6[3413006]: segfault at 8 ip 00007f54163db875 sp 00007ffc19757c50 error 4 in ld-2.28.so[7f54163ba000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 3431257:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0xbce4:0x0] mdc close failed: rc = -13
Lustre: 3093741:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 3093741:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
8[3435059]: segfault at 0 ip 000056500f16454e sp 00007ffc841b5c88 error 6 in 8[56500f161000+7000]
Code: e9 8f f2 ff ff 89 d0 48 8b 5c 24 48 48 85 db 74 2c 84 c0 74 28 0f b6 0b 48 89 d8 84 c9 74 1e 4c 89 fa 4c 29 f8 49 39 d4 76 04 <41> 88 0c 16 48 83 c2 01 0f b6 0c 10 84 c9 75 eb 49 89 d7 4d 39 fc
8[3435170]: segfault at 0 ip 000055a645c8a54e sp 00007ffebf02be78 error 6 in 8[55a645c87000+7000]
Code: e9 8f f2 ff ff 89 d0 48 8b 5c 24 48 48 85 db 74 2c 84 c0 74 28 0f b6 0b 48 89 d8 84 c9 74 1e 4c 89 fa 4c 29 f8 49 39 d4 76 04 <41> 88 0c 16 48 83 c2 01 0f b6 0c 10 84 c9 75 eb 49 89 d7 4d 39 fc
2[3446467]: segfault at 8 ip 00007fa1a2b54875 sp 00007ffd6bef9c60 error 4 in ld-2.28.so[7fa1a2b33000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[3492416]: segfault at 8 ip 00007fa3c6609875 sp 00007ffe3241ac90 error 4 in ld-2.28.so[7fa3c65e8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[3514418]: segfault at 8 ip 00007f1848cae875 sp 00007ffe75a9ea70 error 4 in ld-2.28.so[7f1848c8d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[3521781]: segfault at 5612b6418000 ip 00005612b6418000 sp 00007fff0f8f8f58 error 14 in 5[5612b6618000+1000]
Code: Unable to access opcode bytes at RIP 0x5612b6417fd6.
3[3527705]: segfault at 8 ip 00007f4a7c06f875 sp 00007ffc4e023380 error 4 in ld-2.28.so[7f4a7c04e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 4[3570184] general protection fault ip:55fbae80fb06 sp:7ffd83bdaa98 error:0 in 4[55fbae80a000+7000]
Lustre: 3094003:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 3094003:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 451 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 3094003:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 457 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 3094003:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 457 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 3094003:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 457 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 3094003:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 457 previous similar messages
Lustre: 3094003:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 3094003:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 457 previous similar messages
LustreError: 3602569:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffffa00611aad800: inode [0x200000401:0x12020:0x0] mdc close failed: rc = -13
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
Lustre: 3096343:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
7[3664303]: segfault at 8 ip 00007f11776d1875 sp 00007ffce9a6bfd0 error 4 in ld-2.28.so[7f11776b0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 3093743:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 3093743:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Link to test (2026-08-22 12:43)
racer test 1: racer on clients: centos-110.localnet DURATION=2700
LustreError: 2600820:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 2600820:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 2 PID: 2600820 Comm: ll_sa_2600800 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_statahead_thread+0x14ca/0x15b0 [lustre]
? ll_statahead_by_list+0xd00/0xd00 [lustre]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: 2449150:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e47e73c8000 x1874165479152384/t4294967389(0) o101->0203463b-4735-491e-a529-d8b08171ab6f@0@lo:339/0 lens 376/816 e 0 to 0 dl 1787343549 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 2450011:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 2450011:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2450011:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 2450011:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 2450011:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2450011:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2447182:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2447182:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 2447182:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2447182:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2447182:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2447182:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2447182:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 2447182:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2447182:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2447182:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2447182:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2447182:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2447181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2447181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 2447181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2447181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2447181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 2447181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2447181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2447181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2451205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 543, rollback = 7
Lustre: 2451205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 2451205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2451205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2451205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2451205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/543/0, punch: 0/0/0, quota 4/150/0
Lustre: 2451205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2451205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2451205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2447180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 2447180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2447180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2447180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 2447180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2447180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 2447180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2447180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
10[2464780]: segfault at 0 ip 0000564ad9bb7b47 sp 00007ffd51135330 error 6 in 10[564ad9bb3000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 2451205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2451205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2451205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/78/0
Lustre: 2451205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
13[2468007]: segfault at 8 ip 00007f6ab101c875 sp 00007ffce9716250 error 4 in ld-2.28.so[7f6ab0ffb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 2451205:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 585, rollback = 7
Lustre: 2451205:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2451205:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/585/0, punch: 0/0/0, quota 1/3/0
Lustre: 2451205:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 2451205:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2451205:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 9 previous similar messages
Lustre: 2447180:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2447180:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 23 previous similar messages
Lustre: 2447180:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2447180:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 2447180:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2447180:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 2447180:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 2447180:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 2447180:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2447180:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 23 previous similar messages
Lustre: 2447180:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2447180:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 23 previous similar messages
8[2488361]: segfault at 8 ip 00007f2fb6067875 sp 00007ffecc583600 error 4 in ld-2.28.so[7f2fb6046000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[2489636]: segfault at 8 ip 00007f601aa1b875 sp 00007ffdfa1a1a40 error 4 in ld-2.28.so[7f601a9fa000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 2450011:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2450011:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 39 previous similar messages
Lustre: 2450011:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2450011:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 2450011:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2450011:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 2450011:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 2450011:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 2450011:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2450011:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 39 previous similar messages
Lustre: 2450011:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2450011:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 39 previous similar messages
10[2493927]: segfault at 8 ip 00007f7808ad1875 sp 00007ffdab699fc0 error 4 in ld-2.28.so[7f7808ab0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[2494001]: segfault at 8 ip 00007f7c03d7e875 sp 00007fffc6b05a50 error 4 in ld-2.28.so[7f7c03d5d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 2494001:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e47a2f9f800: inode [0x200000402:0xe01:0x0] mdc close failed: rc = -13
12[2494788]: segfault at 563f51671000 ip 0000563f51671000 sp 00007fff7a27cef0 error 14 in 12[563f51871000+1000]
Code: Unable to access opcode bytes at RIP 0x563f51670fd6.
16[2497764]: segfault at 8 ip 00007feb9af6a875 sp 00007ffedb1c8b30 error 4 in ld-2.28.so[7feb9af49000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[2497806]: segfault at 8 ip 00007f27b1b42875 sp 00007ffe80b52310 error 4 in ld-2.28.so[7f27b1b21000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 3[2500232] trap stack segment ip:562868cdaab4 sp:7ffd7bb16bf0 error:0 in 3[562868cd6000+7000]
traps: 3[2500258] trap stack segment ip:55f42f3d6ab4 sp:7fff35a0ce20 error:0 in 3[55f42f3d2000+7000]
LustreError: 2503589:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e47a2f9f800: inode [0x200000401:0x12f0:0x0] mdc close failed: rc = -13
Lustre: 2452168:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
1[2513590]: segfault at 8 ip 00007f6287ca2875 sp 00007ffefb13ee10 error 4 in ld-2.28.so[7f6287c81000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 2513955:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e47a2f9f800: inode [0x200000401:0x16f3:0x0] mdc close failed: rc = -13
LustreError: 2519897:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e478562f000: inode [0x200000402:0x16bd:0x0] mdc close failed: rc = -13
Lustre: 2483474:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2483474:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 89 previous similar messages
Lustre: 2483474:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2483474:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 2483474:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2483474:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 2483474:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 2483474:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 2483474:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2483474:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 90 previous similar messages
Lustre: 2483474:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2483474:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 90 previous similar messages
17[2533847]: segfault at 8 ip 00007f1985eca875 sp 00007ffeb9016450 error 4 in ld-2.28.so[7f1985ea9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 3[2535366] trap invalid opcode ip:55e4f326166c sp:7ffdfa5ebae8 error:0 in 3[55e4f325c000+7000]
9[2539808]: segfault at 8 ip 00007f5aabdc3875 sp 00007fff329183d0 error 4 in ld-2.28.so[7f5aabda2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: mdt_io00_001: service thread pid 2446919 was inactive for 43.823 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt00_017 state:I stack:0 pid:2454010 ppid:2 flags:0x80004080
Lustre: Skipped 1 previous similar message
task:mdt00_004 state:I stack:0 pid:2449051 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
Lustre: mdt00_014: service thread pid 2450383 was inactive for 43.023 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Call Trace:
__schedule+0x351/0xcb0
task:mdt_io00_001 state:I stack:0 pid:2446919 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
schedule+0xc4/0x180
? woken_wake_function+0x40/0x40
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
mdt_object_lock+0xa1/0x250 [mdt]
mdt_getattr_name_lock+0x26e1/0x32e0 [mdt]
? __req_capsule_get+0x494/0xb30 [ptlrpc]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_check_lock+0x24b/0x4c0 [mdt]
mdt_reint_rename+0x17f7/0x34a0 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
LustreError: 2446895:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff9e483b8a6a00/0x7e627d7690b89752 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x2106:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x7e627d7690b895b5 expref: 1079 pid: 2449367 timeout: 32079 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt00_004: service thread pid 2449051 completed after 100.370s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_017: service thread pid 2454010 completed after 100.386s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_009: service thread pid 2449151 completed after 100.388s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_018: service thread pid 2455955 completed after 100.389s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_013: service thread pid 2450374 completed after 100.389s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_014: service thread pid 2450383 completed after 100.407s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt_io00_001: service thread pid 2446919 completed after 101.223s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_008: service thread pid 2449150 completed after 100.419s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: mdt00_016: service thread pid 2450619 completed after 100.419s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 2449367:0:(ldlm_lockd.c:1509:ldlm_handle_enqueue()) ### lock on destroyed export 00000000bf811df7 ns: mdt-lustre-MDT0000_UUID lock: ffff9e47e57b3c00/0x7e627d7690b8a4f7 lrc: 3/0,0 mode: PR/PR res: [0x200000402:0x1ab1:0x0].0x0 bits 0x13/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x7e627d7690b895c3 expref: 5 pid: 2449367 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt_out00_003: service thread pid 2449367 completed after 101.243s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0000-mdc-ffff9e478562f000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: mdt00_005: service thread pid 2449118 completed after 101.254s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0000-mdc-ffff9e478562f000: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: lustre-MDT0000-mdc-ffff9e478562f000: operation ldlm_enqueue to node 0@lo failed: rc = -107
LustreError: 2552468:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e478562f000: inode [0x200000401:0x1:0x0] mdc close failed: rc = -108
LustreError: 2552546:0:(file.c:6167:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108
Lustre: lustre-MDT0000-mdc-ffff9e478562f000: Connection restored to 0@lo (at 0@lo)
5[2556794]: segfault at 8 ip 00007f2515f7d875 sp 00007ffc12c08c50 error 4 in ld-2.28.so[7f2515f5c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[2558330]: segfault at 8 ip 00007f0d85376875 sp 00007ffcae343570 error 4 in ld-2.28.so[7f0d85355000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 2447181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 2447181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 101 previous similar messages
Lustre: 2447181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 2447181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 101 previous similar messages
Lustre: 2447181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 2447181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 101 previous similar messages
Lustre: 2447181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 2447181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 101 previous similar messages
Lustre: 2447181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 2447181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 101 previous similar messages
Lustre: 2447181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 2447181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 101 previous similar messages
19[2561379]: segfault at 8 ip 00007fef8a5e3875 sp 00007ffe63c937f0 error 4 in ld-2.28.so[7fef8a5c2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[2576065]: segfault at 0 ip 000055f492473b47 sp 00007ffd1f791340 error 6 in 17[55f49246f000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 2446919:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
1[2598717]: segfault at ffffffffffffff84 ip 00005606bee7f019 sp 00007ffe97e70c48 error 7 in 1[5606bee7c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 e9 6b f7 ff <ff> 49 83 c7 04 41 89 c0 45 31 ed bb 30 00 00 00 e9 e8 f5 ff ff 85
Link to test (2026-08-21 20:32)
sanity test 154g: various llapi FID tests
LustreError: 695265:0:(statahead.c:582:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 695265:0:(statahead.c:582:ll_sax_put()) LBUG
CPU: 1 UID: 0 PID: 695265 Comm: ll_sa_694198 Kdump: loaded Tainted: G W OE 7.0.0-22-generic #22-Ubuntu PREEMPT(lazy)
Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
show_stack+0x49/0x60
dump_stack_lvl+0x5f/0x90
dump_stack+0x10/0x18
lbug_with_loc.cold+0x5/0x50 [libcfs]
ll_statahead_thread+0x1ba1/0x1f40 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? _raw_spin_unlock_irqrestore+0x11/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? _raw_spin_unlock_irqrestore+0x11/0x60
? __pfx_ll_statahead_thread+0x10/0x10 [lustre]
kthread+0xf7/0x130
? __pfx_kthread+0x10/0x10
ret_from_fork+0x195/0x2a0
? __pfx_kthread+0x10/0x10
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Link to test (2026-07-28 21:45)
racer test 2: racer rename: trevis-154vm171.trevis.whamcloud.com,trevis-154vm172 DURATION=300
LustreError: 505350:0:(statahead.c:583:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 505350:0:(statahead.c:583:ll_sax_put()) LBUG
CPU: 0 PID: 505350 Comm: ll_sa_505061 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
ll_statahead_thread+0x9b8/0xab0 [lustre]
? __pfx_ll_statahead_thread+0x10/0x10 [lustre]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
LustreError: 146399:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep sleep [0x200000be9:0x1870:0x0] inode@0000000000000000: rc = -5
LustreError: 307660:0:(llite_nfs.c:391:ll_dir_get_parent_fid()) lustre: failure inode [0x240000bec:0x27e2:0x0] get parent: rc = -116
Lustre: dir [0x280000be9:0x4433:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 155 previous similar messages
LustreError: 479696:0:(llite_nfs.c:391:ll_dir_get_parent_fid()) lustre: failure inode [0x280000be9:0x4ddc:0x0] get parent: rc = -116
Link to test (2026-04-16 08:10)
racer test 2: racer rename: onyx-55vm4.onyx.whamcloud.com,onyx-55vm5 DURATION=300
LustreError: 663511:0:(statahead.c:583:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 663511:0:(statahead.c:583:ll_sax_put()) LBUG
CPU: 0 PID: 663511 Comm: ll_sa_663369 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.53.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
ll_statahead_thread+0x1dcd/0x1f60 [lustre]
? ll_statahead_interpret+0x440/0x440 [lustre]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
LustreError: 569053:0:(lcommon_cl.c:188:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000be8:0x15da:0x0]: rc = -5
LustreError: 569053:0:(lcommon_cl.c:188:cl_file_inode_init()) Skipped 75 previous similar messages
LustreError: 569053:0:(llite_lib.c:3769:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 569053:0:(llite_lib.c:3769:ll_prep_inode()) Skipped 75 previous similar messages
LustreError: 114825:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 4 [0x200000be8:0x15e9:0x0] inode@0000000000000000: rc = -5
LustreError: 73530:0:(statahead.c:825:ll_statahead_interpret_work()) lustre: failed to prep 10 [0x240000bea:0x10b6:0x0] inode@0000000000000000: rc = -5
LustreError: 73530:0:(statahead.c:825:ll_statahead_interpret_work()) Skipped 1 previous similar message
Link to test (2025-06-19 06:53)
racer test 2: racer rename: onyx-144vm4.onyx.whamcloud.com,onyx-144vm5 DURATION=300
LustreError: 1035596:0:(statahead.c:583:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 1035596:0:(statahead.c:583:ll_sax_put()) LBUG
CPU: 1 PID: 1035596 Comm: ll_sa_1035447 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.31.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
ll_statahead_thread+0xc48/0xde0 [lustre]
? __pfx_ll_statahead_thread+0x10/0x10 [lustre]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
LustreError: 783413:0:(lov_object.c:1340:lov_layout_change()) lustre-clilov-ffff8a1234c03800: cannot apply new layout on [0x240000be8:0x953:0x0] : rc = -5
LustreError: 783413:0:(lov_object.c:1340:lov_layout_change()) Skipped 10 previous similar messages
LustreError: 787641:0:(statahead.c:2446:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
LustreError: 787677:0:(lcommon_cl.c:178:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be7:0x884:0x0]: rc = -5
LustreError: 787677:0:(lcommon_cl.c:178:cl_file_inode_init()) Skipped 200 previous similar messages
LustreError: 784269:0:(llite_lib.c:3694:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 784269:0:(llite_lib.c:3694:ll_prep_inode()) Skipped 201 previous similar messages
Autotest: Test running for 255 minutes (lustre-reviews_review-dne-part-9_112146.37)
LustreError: 907429:0:(statahead.c:2446:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
Lustre: dir [0x2c0000be7:0x2fc1:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
Link to test (2025-04-01 09:48)
racer test 2: racer rename: onyx-52vm1.onyx.whamcloud.com,onyx-52vm2 DURATION=300
LustreError: 566701:0:(statahead.c:583:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 566701:0:(statahead.c:583:ll_sax_put()) LBUG
CPU: 0 PID: 566701 Comm: ll_sa_566377 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.31.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
ll_statahead_thread+0xc48/0xde0 [lustre]
? __pfx_ll_statahead_thread+0x10/0x10 [lustre]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_MAX_MB=0 RACER_ENABLE_FLR=false RACER_ENABLE_DOM=false RACER_ENABLE_SEL=false RACER_ENABLE_MIGRATION=false RACER_MAX_CLEANUP_WAIT= RACER_EXTRA="" RACER_EXTRA_LAYOUT="" RACER_PROGS=dir_c
LustreError: 381285:0:(lcommon_cl.c:178:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000be9:0x1091:0x0]: rc = -5
LustreError: 381285:0:(lcommon_cl.c:178:cl_file_inode_init()) Skipped 121 previous similar messages
LustreError: 381285:0:(llite_lib.c:3691:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 381285:0:(llite_lib.c:3691:ll_prep_inode()) Skipped 126 previous similar messages
LustreError: 381369:0:(lov_object.c:1340:lov_layout_change()) lustre-clilov-ffff8a6d497f1000: cannot apply new layout on [0x280000be7:0xccf:0x0] : rc = -5
LustreError: 381369:0:(lov_object.c:1340:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 503609:0:(llite_nfs.c:429:ll_dir_get_parent_fid()) lustre: failure inode [0x240000beb:0x2644:0x0] get parent: rc = -116
Link to test (2025-03-22 00:57)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 273117:0:(statahead.c:580:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 273117:0:(statahead.c:580:ll_sax_put()) LBUG
Pid: 273117, comm: ll_sa_273114 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff8c02ed50d800: disconnect after 20s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff8c02ed50d800: disconnect after 21s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-12-05 09:01)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 144233:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 144233:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 144233, comm: ll_sa_144232 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
NOT correct pid: sai ffff941f60605400 pid=143580:144901
NOT correct pid: sai ffff941f535fdc00 pid=144232:144901
NOT correct pid: sai ffff941f535fdc00 pid=144232:144901
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff941f45e01000: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff941f45e01000: disconnect after 24s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-11-22 06:54)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 272822:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 272822:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 272822, comm: ll_sa_272166 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff8dfaf605f000: disconnect after 20s idle
Lustre: lustre-OST0002-osc-ffff8dfaf605f000: disconnect after 22s idle
Link to test (2023-11-22 02:47)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 294153:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 294153:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 294153, comm: ll_sa_294150 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff93c95f717000: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff93c95f717000: disconnect after 23s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-11-20 06:47)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 273082:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 273082:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 273082, comm: ll_sa_272425 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff88caab370800: disconnect after 24s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff88caab370800: disconnect after 21s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-11-20 06:42)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 303279:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 303279:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 303279, comm: ll_sa_303276 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0002-osc-ffff9cd04531a000: disconnect after 20s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0001-osc-ffff9cd04531a000: disconnect after 22s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-11-17 09:25)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 273347:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 273347:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 273347, comm: ll_sa_273344 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f36/0x2050 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0002-osc-ffff8c28ae841000: disconnect after 24s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0001-osc-ffff8c28ae841000: disconnect after 22s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-11-17 08:40)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 269094:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 269094:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 269094, comm: ll_sa_269091 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Thu Jul 20 08:38:27 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1ecf/0x2030 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff9e4b3603a800: disconnect after 23s idle
Lustre: Skipped 3 previous similar messages
Lustre: lustre-OST0002-osc-ffff9e4b3603a800: disconnect after 21s idle
Link to test (2023-10-20 19:19)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 289454:0:(statahead.c:577:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 289454:0:(statahead.c:577:ll_sax_put()) LBUG
Pid: 289454, comm: ll_sa_288810 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Thu Jul 20 08:38:27 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1ecf/0x2030 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff9446b5b78000: disconnect after 20s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff9446b5b78000: disconnect after 22s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-10-20 19:04)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 269188:0:(statahead.c:578:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 269188:0:(statahead.c:578:ll_sax_put()) LBUG
Pid: 269188, comm: ll_sa_269185 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Thu Jul 20 08:38:27 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f21/0x2070 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff8b24c3a7c800: disconnect after 25s idle
Lustre: lustre-OST0002-osc-ffff8b24c3a7c800: disconnect after 21s idle
Link to test (2023-10-19 10:34)
racer test 1: racer on clients: onyx-67vm3.onyx.whamcloud.com,onyx-67vm4 DURATION=300
LustreError: 277914:0:(statahead.c:578:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 277914:0:(statahead.c:578:ll_sax_put()) LBUG
Pid: 277914, comm: ll_sa_277762 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Thu Jul 20 08:38:27 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f21/0x2070 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Link to test (2023-10-19 08:54)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 270804:0:(statahead.c:580:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 270804:0:(statahead.c:580:ll_sax_put()) LBUG
Pid: 270804, comm: ll_sa_270160 4.18.0-477.21.1.el8_8.x86_64 #1 SMP Thu Jul 20 08:38:27 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f21/0x2070 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff8d9e46124000: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff8d9e46124000: disconnect after 24s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-10-07 11:58)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 288733:0:(statahead.c:580:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 288733:0:(statahead.c:580:ll_sax_put()) LBUG
Pid: 288733, comm: ll_sa_288730 4.18.0-477.15.1.el8_8.x86_64 #1 SMP Fri Jun 2 08:27:19 EDT 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f21/0x2070 [lustre]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff8ccf82b15000: disconnect after 21s idle
Lustre: lustre-OST0002-osc-ffff8ccf82b15000: disconnect after 20s idle
Link to test (2023-09-12 08:29)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 276908:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 276908:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 276908, comm: ll_sa_276905 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f18/0x2090 [lustre]
[<0>] kthread+0x10b/0x130
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff8ed922402000: disconnect after 22s idle
Lustre: lustre-OST0002-osc-ffff8ed922402000: disconnect after 22s idle
Link to test (2023-07-07 09:38)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 277100:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 277100:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 277100, comm: ll_sa_276448 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f18/0x2090 [lustre]
[<0>] kthread+0x10b/0x130
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff9662454e1800: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff9662454e1800: disconnect after 22s idle
Lustre: Skipped 1 previous similar message
Link to test (2023-07-06 15:44)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 253987:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 253987:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 253987, comm: ll_sa_253984 4.18.0-425.10.1.el8_7.x86_64 #1 SMP Wed Dec 14 16:00:01 EST 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f18/0x2090 [lustre]
[<0>] kthread+0x10b/0x130
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff91e442ccc800: disconnect after 23s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0002-osc-ffff91e442ccc800: disconnect after 20s idle
Link to test (2023-07-06 15:09)
parallel-scale-nfsv4 test racer_on_nfs: racer on NFS client
LustreError: 40529:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 40529:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 40529, comm: ll_sa_21450 4.18.0-425.10.1.el8_lustre.x86_64 #1 SMP Wed Apr 5 05:10:27 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f14/0x2090 [lustre]
[<0>] kthread+0x10b/0x130
[<0>] ret_from_fork+0x35/0x40
Link to test (2023-05-04 11:31)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 118322:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 118322:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 118322, comm: ll_sa_118321 4.18.0-348.7.1.el8_5.x86_64 #1 SMP Wed Dec 22 13:25:12 UTC 2021
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0x90 [libcfs]
[<0>] lbug_with_loc+0x43/0x80 [libcfs]
[<0>] ll_statahead_thread+0x1f2d/0x20b0 [lustre]
[<0>] kthread+0x116/0x130
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff9eb583e01000: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0003-osc-ffff9eb583e01000: disconnect after 21s idle
Lustre: Skipped 1 previous similar message
Link to test (2022-12-26 05:10)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 131541:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 131541:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 131541, comm: ll_sa_131538 4.18.0-372.19.1.el8_6.x86_64 #1 SMP Mon Jul 18 11:14:02 EDT 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f1e/0x20a0 [lustre]
[<0>] kthread+0x10a/0x120
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0001-osc-ffff9d4ec47a0000: disconnect after 22s idle
Lustre: lustre-OST0002-osc-ffff9d4ec47a0000: disconnect after 24s idle
Link to test (2022-12-26 05:00)
racer test 1: racer on clients: onyx-130vm5.onyx.whamcloud.com,onyx-130vm6 DURATION=300
LustreError: 25289:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 25289:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 25289, comm: ll_sa_25103 4.18.0-372.19.1.el8_6.x86_64 #1 SMP Mon Jul 18 11:14:02 EDT 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f1e/0x20a0 [lustre]
[<0>] kthread+0x10a/0x120
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MAX_CLEANUP_WAIT= RACER_ENABLE_SE
Link to test (2022-12-08 05:51)
sanity-lfsck test 18d: Find out orphan OST-object and repair it (4)
LustreError: 131769:0:(statahead.c:576:ll_sax_put()) ASSERTION( list_empty(&ctx->sax_sai_list) ) failed:
LustreError: 131769:0:(statahead.c:576:ll_sax_put()) LBUG
Pid: 131769, comm: ll_sa_131768 4.18.0-372.19.1.el8_6.x86_64 #1 SMP Mon Jul 18 11:14:02 EDT 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ll_statahead_thread+0x1f1e/0x20a0 [lustre]
[<0>] kthread+0x10a/0x120
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: lustre-OST0002-osc-ffff99ae04905800: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
Lustre: lustre-OST0001-osc-ffff99ae04905800: disconnect after 22s idle
Lustre: Skipped 1 previous similar message
Link to test (2022-12-03 03:41)
Return to new crashes list