LustreError: 477444:0:(statahead.c:1728:ll_deauthorize_statahead()) ASSERTION( lli->lli_stat_pid != 0 ) failed:
LustreError: 477444:0:(statahead.c:1728:ll_deauthorize_statahead()) LBUG
CPU: 11 PID: 477444 Comm: lfs Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_deauthorize_statahead+0x431/0x570 [lustre]
ll_file_release+0x4d0/0xcc0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f31c67bf7c5

Lustre: 11408:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff935e39e85b00 x1879863468293504/t4294968093(0) o101->1475135d-64a4-46ac-952c-99fdcef038c8@0@lo:631/0 lens 376/840 e 0 to 0 dl 1792777576 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11980:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff935e64917000 x1879863468345344/t4294967912(0) o101->226f0255-4721-467e-86cd-0b98c9d285f3@0@lo:632/0 lens 376/840 e 0 to 0 dl 1792777577 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
7[16663]: segfault at 8 ip 00007fc9c1094875 sp 00007ffd69516930 error 4 in ld-2.28.so[7fc9c1073000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[19966]: segfault at 8 ip 00007effbbc0c875 sp 00007ffd2416bc40 error 4 in ld-2.28.so[7effbbbeb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 12254:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x2e:0x0] ACL: rc = -2
Lustre: 11867:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff935e60da5780 x1879863477168896/t4294971729(0) o101->226f0255-4721-467e-86cd-0b98c9d285f3@0@lo:678/0 lens 376/864 e 0 to 0 dl 1792777623 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
5[33596]: segfault at 8 ip 00007efcd5d2a875 sp 00007ffe08dbc6b0 error 4 in ld-2.28.so[7efcd5d09000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[34854]: segfault at 1b30 ip 0000000000001b30 sp 00007ffc0777bcb0 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
9[45032]: segfault at 8 ip 00007f6970d83875 sp 00007ffeec9d65e0 error 4 in ld-2.28.so[7f6970d62000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[48361]: segfault at 564c7c8d2f24 ip 0000564c7c8d2f24 sp 00007ffd34ce2ed8 error 7 in 9[564c7c8ce000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 16426:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x4ec:0x0] ACL: rc = -2
19[53466]: segfault at 8 ip 00007fc0f15ea875 sp 00007ffda816a280 error 4 in ld-2.28.so[7fc0f15c9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[53642]: segfault at 0 ip 000055e44e91fb47 sp 00007ffea7a26080 error 6 in 18[55e44e91b000+7000]
18[54034]: segfault at 0 ip 000055e571df3b47 sp 00007fff101f6520 error 6 in 18[55e571def000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
19[54889]: segfault at 8 ip 00007f3573bc0875 sp 00007fffe803f5f0 error 4 in ld-2.28.so[7f3573b9f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 56239:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935dcbc3b000: inode [0x200000404:0x660:0x0] mdc close failed: rc = -13
8[59065]: segfault at 8 ip 00007f57892aa875 sp 00007ffec243a0c0 error 4 in ld-2.28.so[7f5789289000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[61418]: segfault at 8 ip 00007fcc665f1875 sp 00007ffdfb24fcf0 error 4 in ld-2.28.so[7fcc665d0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[63707]: segfault at 8 ip 00007f0bfd64a875 sp 00007fff33b1ae10 error 4 in ld-2.28.so[7f0bfd629000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 14971:0:(osd_io.c:1017:osd_grow_blocksize()) object [0x380000402:0xd4:0x0]: change block size4096 -> 1048576 error rc = -95
LustreError: 69993:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935e289cd000: inode [0x200000404:0x7ed:0x0] mdc close failed: rc = -13
12[70732]: segfault at 8 ip 00007f4b65243875 sp 00007fff36451240 error 4 in ld-2.28.so[7f4b65222000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[71799]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[71799]: Failed to get EXE, ignoring: No such process
LustreError: 12060:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x662:0x0] ACL: rc = -2
LustreError: 12060:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 2 previous similar messages
hrtimer: interrupt took 2329461 ns
3[73663]: segfault at 8 ip 00007f0298a4e875 sp 00007ffd32c8a9f0 error 4 in ld-2.28.so[7f0298a2d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 21760:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x773:0x0] ACL: rc = -2
LustreError: 21760:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 1 previous similar message
14[80288]: segfault at 8 ip 00007fa98fb59875 sp 00007ffdca4b2ce0 error 4 in ld-2.28.so[7fa98fb38000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[80572]: segfault at 0 ip 00005652f93d6b47 sp 00007ffcd74c0ff0 error 6 in 16[5652f93d2000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[80256]: segfault at 55f4cfb77000 ip 000055f4cfb77000 sp 00007fffb7129b80 error 14 in 0[55f4cfd77000+1000]
Code: Unable to access opcode bytes at RIP 0x55f4cfb76fd6.
0[81608]: segfault at 558324ba8000 ip 0000558324ba8000 sp 00007ffee01abbd0 error 14 in 0[558324da8000+1000]
Code: Unable to access opcode bytes at RIP 0x558324ba7fd6.
10[81949]: segfault at 8 ip 00007f68eb0d7875 sp 00007fff746ec570 error 4 in ld-2.28.so[7f68eb0b6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 14972:0:(osd_io.c:1017:osd_grow_blocksize()) object [0x380000401:0xca:0x0]: change block size4096 -> 1048576 error rc = -95
12[104323]: segfault at 0 ip 000056072b0478d0 sp 00007ffc76d68588 error 6 in 12[56072b043000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[115985]: segfault at 8 ip 00007fd738fb4875 sp 00007ffdbd3b21c0 error 4 in ld-2.28.so[7fd738f93000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[116314]: segfault at 0 ip 000055e3b5e52b47 sp 00007ffc01bbd1d0 error 6 in 10[55e3b5e4e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7[116440]: segfault at 8 ip 00007fd8141cc875 sp 00007fff35d088c0 error 4 in ld-2.28.so[7fd8141ab000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 116440:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935e289cd000: inode [0x200000404:0xeb9:0x0] mdc close failed: rc = -13
5[117663]: segfault at 8 ip 00007f7d457e8875 sp 00007fff26b32800 error 4 in ld-2.28.so[7f7d457c7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 11566:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0xafe:0x0] ACL: rc = -2
10[123038]: segfault at 8 ip 00007fecaa6e7875 sp 00007ffde9559bf0 error 4 in ld-2.28.so[7fecaa6c6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 128638:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935dcbc3b000: inode [0x280000403:0xbc8:0x0] mdc close failed: rc = -13
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000404:0xbb2:0x0]/0xa): rc = 0
7[132012]: segfault at 8 ip 00007f8af7f0d875 sp 00007ffc70d42b60 error 4 in ld-2.28.so[7f8af7eec000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 135658:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935dcbc3b000: inode [0x280000403:0xbc8:0x0] mdc close failed: rc = -13
17[136329]: segfault at 8 ip 00007f5c9c341875 sp 00007ffdd93b9b00 error 4 in ld-2.28.so[7f5c9c320000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[140851]: segfault at 8 ip 00007f226d3a5875 sp 00007ffe94959170 error 4 in ld-2.28.so[7f226d384000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 12077:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x12:0x0] ACL: rc = -2
18[159522]: segfault at 0 ip 000055ad3c738200 sp 00007ffff6a6eca8 error 6 in 18[55ad3c736000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
1[162472]: segfault at 0 ip 0000557523dcab47 sp 00007ffd122d1e30 error 6 in 1[557523dc6000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ODEBUG: object 00000000eceb7844 is on stack 00000000ef39d8c1, but NOT annotated.
WARNING: CPU: 8 PID: 12002 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon i2c_piix4 pcspkr rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 8 PID: 12002 Comm: mdt00_023 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: be 9f 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffba8162fdf320 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffba8162fdf468 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff935ff241e5a8 RDI: ffff935ff241e5a8
RBP: ffffba8162fdf370 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffba8162fdf0f0 R12: ffffffffa16c93a8
R13: ffffffffa0305aa0 R14: 0000000000043fe0 R15: ffff935dfd295168
FS: 0000000000000000(0000) GS:ffff935ff2400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f55f66173e0 CR3: 000000031e416000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x110c/0x2620 [lod]
? slab_free_freelist_hook.isra.49+0x111/0x230
lod_comp_prep_create+0x1780/0x1c30 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? __osd_sa_xattr_get.isra.8+0x74/0x290 [osd_zfs]
lod_declare_create+0x237/0xc30 [lod]
? osd_xattr_get+0x2d7/0x8d0 [osd_zfs]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x6d7/0x2550 [mdd]
mdt_reint_open+0x3338/0x3c40 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6ad/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.50+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 4e156b2f56c8e10c ]---
LustreError: 167919:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935e289cd000: inode [0x240000404:0x1037:0x0] mdc close failed: rc = -13
4[170718]: segfault at 8 ip 00007ff68344c875 sp 00007fffa7cdc3d0 error 4 in ld-2.28.so[7ff68342b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 12002:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x17d6:0x0] ACL: rc = -2
14[182623]: segfault at 8 ip 00007ff4b2b59875 sp 00007ffcf807e660 error 4 in ld-2.28.so[7ff4b2b38000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[189768]: segfault at 74 ip 00005604b9fc2309 sp 00007fff70248098 error 4 in 6[5604b9fbd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 69 6e 66 6f 20 27 28 <63> 6f 72 65 75 74 69 6c 73 29 20 25 73 25 73 27 0a 00 00 00 00 00
7[196269]: segfault at 8 ip 00007fecde23d875 sp 00007ffed3ea5730 error 4 in ld-2.28.so[7fecde21c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 3[200402] trap invalid opcode ip:55d5c9bb46e1 sp:7ffc3d4d2b50 error:0 in 3[55d5c9bae000+7000]
16[202927]: segfault at 8 ip 00007fc66c1cf875 sp 00007ffe4ed595b0 error 4 in ld-2.28.so[7fc66c1ae000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[213335]: segfault at 8 ip 00007fbb0b734875 sp 00007fffff826a40 error 4 in ld-2.28.so[7fbb0b713000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 218342:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935dcbc3b000: inode [0x200000403:0x1fda:0x0] mdc close failed: rc = -13
LustreError: 15083:0:(osd_io.c:1017:osd_grow_blocksize()) object [0x2c0000402:0x31d:0x0]: change block size262144 -> 1048576 error rc = -95
8[227008]: segfault at 8 ip 00007fc16e111875 sp 00007ffff981af40 error 4 in ld-2.28.so[7fc16e0f0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[233322]: segfault at 8 ip 00007f6096cb2875 sp 00007fffbf098b80 error 4 in ld-2.28.so[7f6096c91000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[238735]: segfault at 8 ip 00007f81a17a4875 sp 00007ffd9ab9c620 error 4 in ld-2.28.so[7f81a1783000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[252448]: segfault at 0 ip 0000564555ae05c0 sp 00007fffd5fd3470 error 6 in 6[564555adf000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
12[255494]: segfault at 8 ip 00007f42d19d2875 sp 00007fffe0a37130 error 4 in ld-2.28.so[7f42d19b1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[265525]: segfault at 0 ip 000055fd40c21b47 sp 00007ffdda472730 error 6 in 14[55fd40c1d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 228789:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x16d2:0x0] ACL: rc = -2
LustreError: 228789:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 1 previous similar message
10[279181]: segfault at 8 ip 00007fb727d37875 sp 00007ffc68228880 error 4 in ld-2.28.so[7fb727d16000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[279838]: segfault at 8 ip 00007f195529f875 sp 00007ffc053bbab0 error 4 in ld-2.28.so[7f195527e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 281226:0:(statahead.c:2479:start_statahead_thread()) lustre: cannot start ll_sa_281226 thread for 'racer1': rc = -4
LustreError: 281226:0:(file.c:6392:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x1d82:0x0] error: rc = -4
9[285953]: segfault at 8 ip 00007f4d78b1b875 sp 00007ffdb34cfe10 error 4 in ld-2.28.so[7f4d78afa000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[285988]: segfault at 8 ip 00007fa6924da875 sp 00007fff5167d3e0 error 4 in ld-2.28.so[7fa6924b9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 12008:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x2921:0x0] ACL: rc = -2
ODEBUG: object 00000000947abea6 is on stack 00000000dcf0e014, but NOT annotated.
WARNING: CPU: 15 PID: 21826 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon i2c_piix4 pcspkr rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 15 PID: 21826 Comm: mdt00_062 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: be 9f 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffba815839b320 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffba815839b468 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff935ff25de5a8 RDI: ffff935ff25de5a8
RBP: ffffba815839b370 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffba815839b0f0 R12: ffffffffa16ef7c8
R13: ffffffffa0305aa0 R14: 000000000006a400 R15: ffff935e2e444190
FS: 0000000000000000(0000) GS:ffff935ff25c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000602064 CR3: 000000013d043000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x110c/0x2620 [lod]
? slab_post_alloc_hook+0x6d/0x390
? __check_object_size+0x10e/0x26d
lod_comp_prep_create+0x1780/0x1c30 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? __osd_sa_xattr_get.isra.8+0x74/0x290 [osd_zfs]
lod_declare_create+0x237/0xc30 [lod]
? osd_xattr_get+0x2d7/0x8d0 [osd_zfs]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x6d7/0x2550 [mdd]
mdt_reint_open+0x3338/0x3c40 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6ad/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.50+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 4e156b2f56c8e10d ]---
LustreError: 298114:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935e289cd000: inode [0x280000403:0x1c1f:0x0] mdc close failed: rc = -13
8[313865]: segfault at 8 ip 00007f6bff7f2875 sp 00007ffd696207c0 error 4 in ld-2.28.so[7f6bff7d1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[315298]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[315298]: Failed to get EXE, ignoring: No such process
12[319232]: segfault at 8 ip 00007f2890f5a875 sp 00007ffe09e59460 error 4 in ld-2.28.so[7f2890f39000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[323401]: segfault at 561bd117e000 ip 0000561bd117e000 sp 00007fff3635f728 error 14 in 5[561bd137e000+1000]
Code: Unable to access opcode bytes at RIP 0x561bd117dfd6.
0[323995]: segfault at 0 ip 00005565ff2a8a88 sp 00007fff353c87e0 error 6 in 0[5565ff2a7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
15[325346]: segfault at 8 ip 00007f3c7227b875 sp 00007ffc11d019b0 error 4 in ld-2.28.so[7f3c7225a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[325296]: segfault at 0 ip 00005624acaccb47 sp 00007ffcc4fa30a0 error 6 in 5[5624acac8000+7000]
Code: Unable to access opcode bytes at RIP 0x5624acaccb1d.
traps: 4[328251] trap invalid opcode ip:5628b780a5ce sp:7ffd944b6bf8 error:0 in 4[5628b7805000+7000]
6[339365]: segfault at 1410 ip 0000000000001410 sp 00007ffd29467368 error 14 in 6[563244892000+7000]
Code: Unable to access opcode bytes at RIP 0x13e6.
LustreError: 93809:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000402:0x3b:0x0] ACL: rc = -2
LustreError: 93809:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 4 previous similar messages
9[342145]: segfault at 0 ip 00005627a6326b47 sp 00007ffe6931aad0 error 6 in 12[5627a6322000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[342575]: segfault at 8 ip 00007ff14f938875 sp 00007fffcfcb8400 error 4 in ld-2.28.so[7ff14f917000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 16[344028] trap invalid opcode ip:5616ae24ebca sp:7fffd2a8ca10 error:0 in 16[5616ae248000+7000]
3[379438]: segfault at 8 ip 00007f6d92c93875 sp 00007ffcc0bf6e10 error 4 in ld-2.28.so[7f6d92c72000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[384634]: segfault at 8 ip 00007fe3404ad875 sp 00007fffd38b8a40 error 4 in ld-2.28.so[7fe34048c000+2f000]
1[384492]: segfault at 8 ip 00007fbbc55c7875 sp 00007ffce0423180 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7fbbc55a6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[385503]: segfault at 8 ip 00007f06153fa875 sp 00007ffe1e176800 error 4 in ld-2.28.so[7f06153d9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 383500:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
LustreError: 393519:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff935e289cd000: inode [0x200000403:0x3773:0x0] mdc close failed: rc = -13
LustreError: 393519:0:(file.c:252:ll_close_inode_openhandle()) Skipped 2 previous similar messages
4[406328]: segfault at 8 ip 00007f86f50b8875 sp 00007fffbf5ea430 error 4 in ld-2.28.so[7f86f5097000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[410103]: segfault at 0 ip 000055f2e5b33b47 sp 00007ffdc1fd8190 error 6 in 11[55f2e5b2f000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 93838:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000401:0x24:0x0] ACL: rc = -2
LustreError: 93838:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 2 previous similar messages
10[426676]: segfault at 8 ip 00007f95076cb875 sp 00007ffe67552b60 error 4 in ld-2.28.so[7f95076aa000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[450277]: segfault at 8 ip 00007f964a3c8875 sp 00007ffc08ac2f60 error 4 in ld-2.28.so[7f964a3a7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[453345]: segfault at 8 ip 00007feb75961875 sp 00007fff4c15e880 error 4 in ld-2.28.so[7feb75940000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[453532]: segfault at 8 ip 00007f4b4e791875 sp 00007ffd75f3ff30 error 4 in ld-2.28.so[7f4b4e770000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[457625]: segfault at 8 ip 00007f42c9d43875 sp 00007ffd51f0f520 error 4 in ld-2.28.so[7f42c9d22000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[463877]: segfault at 8 ip 00007f3269255875 sp 00007ffcb899a9b0 error 4 in ld-2.28.so[7f3269234000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[464079]: segfault at 8 ip 00007f5dd7d0f875 sp 00007ffd9e71d7a0 error 4 in ld-2.28.so[7f5dd7cee000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[464652]: segfault at 0 ip 0000558e762fdb47 sp 00007fffebce9ed0 error 6 in 13[558e762f9000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ODEBUG: object 00000000c0f36901 is on stack 000000003e028ed8, but NOT annotated.
WARNING: CPU: 5 PID: 18011 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon i2c_piix4 pcspkr rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 5 PID: 18011 Comm: mdt00_055 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: be 9f 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffba815259f320 EFLAGS: 00010006
RAX: 0000000000000050 RBX: ffffba815259f468 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff935ff235e5a8 RDI: ffff935ff235e5a8
RBP: ffffba815259f370 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffba815259f0f0 R12: ffffffffa16e2668
R13: ffffffffa0305aa0 R14: 000000000005d2a0 R15: ffff935e826b5500
FS: 0000000000000000(0000) GS:ffff935ff2340000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005614c8851608 CR3: 000000015d5f6000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x110c/0x2620 [lod]
? slab_free_freelist_hook.isra.49+0x111/0x230
lod_comp_prep_create+0x1780/0x1c30 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
lod_declare_create+0x237/0xc30 [lod]
? osd_xattr_get+0x2d7/0x8d0 [osd_zfs]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x6d7/0x2550 [mdd]
mdt_reint_open+0x3338/0x3c40 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6ad/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.50+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 4e156b2f56c8e10e ]---
traps: 1[475500] trap invalid opcode ip:561b7cb286fe sp:7ffd713ec798 error:0 in 1[561b7cb23000+7000]
11[476197]: segfault at 8 ip 00007fd2176d3875 sp 00007ffc364dd9e0 error 4 in ld-2.28.so[7fd2176b2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[476172]: segfault at 8 ip 00007f5d0f68b875 sp 00007ffdc0c5f460 error 4 in ld-2.28.so[7f5d0f66a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48

LustreError: 143243:0:(statahead.c:1728:ll_deauthorize_statahead()) ASSERTION( lli->lli_stat_pid != 0 ) failed:
LustreError: 143243:0:(statahead.c:1728:ll_deauthorize_statahead()) LBUG
CPU: 6 PID: 143243 Comm: rm Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_deauthorize_statahead+0x431/0x570 [lustre]
ll_file_release+0x4d0/0xcc0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f2630324cd8

Lustre: 8103:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9bc5c640f000 x1879223894777856/t4294970027(0) o101->c6dd9977-7640-4471-a098-01720ea95b4e@0@lo:723/0 lens 376/864 e 0 to 0 dl 1792167628 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 5528:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 5528:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0
Lustre: 5528:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 17/17/1, xattr_set: 26/520/0
Lustre: 5528:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 24/168/0, punch: 0/0/0, quota 7/249/0
Lustre: 5528:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 7/117/0, delete: 3/6/0
Lustre: 5528:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0
Lustre: 6246:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6246:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6246:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6246:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6246:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6246:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 6246:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6246:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6246:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6246:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6246:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 10431:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10431:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10431:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10431:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10431:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10431:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 6246:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6246:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 6246:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6246:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6246:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6246:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6246:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6246:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6246:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6246:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 6246:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6246:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 10431:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 531, rollback = 7
Lustre: 10431:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10431:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10431:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/531/0, punch: 0/0/0, quota 1/3/0
Lustre: 10431:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10431:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10431:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10431:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: ll_ost_io00_001: service thread pid 6242 was inactive for 41.758 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_001 state:R running task stack:0 pid:6242 ppid:2 flags:0x80004080
Call Trace:
? osd_trunc_lock+0x127/0x330 [osd_ldiskfs]
? __might_sleep+0x60/0xc0
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_trans_create+0x7f/0x2b0 [ofd]
? ofd_object_fallocate+0x36b/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
Lustre: 8381:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 8381:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 8381:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8381:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8381:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 8381:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8381:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8381:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8381:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8381:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 8381:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8381:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6241:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 6241:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 9 previous similar messages
Lustre: 6241:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6241:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6241:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6241:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6241:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6241:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6241:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6241:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6241:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6241:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 7 previous similar messages
Lustre: 6246:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6246:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 13 previous similar messages
LustreError: 5504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 0000000098b6a3d2/0xd63ccd2c3a2e7a74 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0xa:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 12288->49151) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd63ccd2c3a2e7a5f expref: 526 pid: 6235 timeout: 188 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3001:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x380:0x0]/ may get corrupted (rc -108)
Lustre: 3002:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x485:0x0]/ may get corrupted (rc -108)
Lustre: 3000:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x447:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_001: service thread pid 6242 completed after 106.189s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: 8381:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 8381:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 8381:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 8381:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 8381:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 8381:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 8381:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 8381:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 19 previous similar messages
Lustre: 8381:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 8381:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 19 previous similar messages
7[34759]: segfault at 8 ip 00007fbb7515d875 sp 00007fff63691930 error 4 in ld-2.28.so[7fbb7513c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 34759:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9bc620200800: inode [0x200000402:0x563:0x0] mdc close failed: rc = -13
Lustre: 6241:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 6241:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 7 previous similar messages
LustreError: 5504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 00000000b085a1d3/0xd63ccd2c3a336f49 lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x22:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 8192->20479) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd63ccd2c3a336efc expref: 48880 pid: 8261 timeout: 221 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff9bc60c7ed000: operation ost_setattr to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff9bc60c7ed000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff9bc60c7ed000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 2998:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x717:0x0]/ may get corrupted (rc -108)
Lustre: 3013:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x615:0x0]// may get corrupted (rc -108)
LustreError: 38064:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0001-osc-ffff9bc60c7ed000: namespace resource [0x280000400:0x22:0x0].0x0 (ffff9bc5c5fb5000) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff9bc60c7ed000: Connection restored to 0@lo (at 0@lo)
traps: 3[40448] trap invalid opcode ip:55a4c2d41c8a sp:2800000b error:0 in 3[55a4c2d3b000+7000]
6[44248]: segfault at 0 ip 00005651dfb0db47 sp 00007fff4a37e660 error 6 in 6[5651dfb09000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 44545:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9bc60c7ed000: inode [0x200000402:0x8fa:0x0] mdc close failed: rc = -13
Lustre: 10507:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10507:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10507:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10507:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10507:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 10507:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10507:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10507:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10507:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10507:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 15 previous similar messages
Lustre: 10431:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10431:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 17 previous similar messages
Lustre: 5528:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 5528:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 5504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000e626e1d3/0xd63ccd2c3a447888 lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x91:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->8191) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd63ccd2c3a447881 expref: 840 pid: 12238 timeout: 336 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3011:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0xcec:0x0]/ may get corrupted (rc -108)
Lustre: 3012:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0xe73:0x0]/ may get corrupted (rc -108)
Lustre: 3011:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0xe05:0x0]/ may get corrupted (rc -108)
Lustre: 3003:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0xdf5:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection restored to 0@lo (at 0@lo)
Lustre: 5529:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 5529:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10184:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 10184:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
INFO: task fallocate:35392 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35392 ppid:7438 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fdf9cae870b
Code: Unable to access opcode bytes at RIP 0x7fdf9cae86e1.
RSP: 002b:00007ffcb1afc3f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdf9cae870b
RDX: 000000000000784b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001c11 R11: 0000000000000246 R12: 00007ffcb1afc618
R13: 0000000000000000 R14: 00007ffcb1afc458 R15: 000000000000784b
INFO: task chmod:35534 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:35534 ppid:7431 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fa8a10e1258
Code: Unable to access opcode bytes at RIP 0x7fa8a10e122e.
RSP: 002b:00007fff5d5d4d38 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa8a10e1258
RDX: 00000000000009b2 RSI: 0000560473779500 RDI: 00000000ffffff9c
RBP: 0000560473779470 R08: 0000000000000000 R09: 00000000000009b2
R10: 0000000000000fff R11: 0000000000000246 R12: 0000560471ee7bb4
R13: 0000560473779500 R14: 000056047377a730 R15: 0000000000008000
Lustre: 6242:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6242:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 6242:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 6242:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 6242:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/585/0, punch: 0/0/0, quota 1/3/0
Lustre: 6242:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 6242:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6242:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 6242:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6242:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 61 previous similar messages
Lustre: 45547:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 45547:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 65 previous similar messages
Lustre: 10184:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 516 < left 520, rollback = 2
Lustre: 10184:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: ll_ost_io00_000: service thread pid 6241 was inactive for 226.604 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_000 state:R stack:0 pid:6241 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? bit_wait+0x90/0x90
schedule+0xc4/0x180
io_schedule+0x26/0x60
bit_wait_io+0x19/0x90
__wait_on_bit+0x83/0xe0
out_of_line_wait_on_bit+0x98/0xc0
? __might_sleep+0x60/0xc0
? _cond_resched+0x21/0x50
? _raw_spin_lock_irqsave+0x67/0x80
? get_page_from_freelist+0x1061/0x20f0
? __ldiskfs_handle_dirty_metadata+0x81/0x2e0 [ldiskfs]
? __ldiskfs_journal_get_write_access+0x64/0xd0 [ldiskfs]
? __alloc_pages_nodemask+0x22a/0x550
? alloc_pages_current+0x9b/0x1b0
? allocate_slab+0x39d/0x840
? new_slab+0x46/0xa0
? ___slab_alloc+0x5ff/0xfa0
? osd_trunc_unlock_all+0xf2/0x1e0 [osd_ldiskfs]
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? __might_sleep+0x60/0xc0
? should_failslab+0x16/0x40
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? __slab_alloc.isra.60+0x38/0x60
? kmem_cache_alloc_trace+0x3cf/0x440
? osd_attr_get+0x280/0x8b0 [osd_ldiskfs]
? __might_sleep+0x60/0xc0
? osd_trans_create+0x180/0x5d0 [osd_ldiskfs]
? ofd_trans_create+0x7f/0x2b0 [ofd]
? ofd_object_fallocate+0x36b/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
hrtimer: interrupt took 4031384 ns
Lustre: 10184:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 10184:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 1 previous similar message
LustreError: 5504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 104s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000007bd27491/0xd63ccd2c3a57147f lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x13c:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 24576->53247) gid 0 flags: 0x60000400020020 nid: 0@lo remote: 0xd63ccd2c3a571478 expref: 1337 pid: 19708 timeout: 468 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0001-osc-ffff9bc60c7ed000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff9bc60c7ed000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0001-osc-ffff9bc60c7ed000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3001:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x160b:0x0]/ may get corrupted (rc -108)
Lustre: 3007:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x1774:0x0]/ may get corrupted (rc -108)
Lustre: 3006:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x16b9:0x0]/ may get corrupted (rc -108)
Lustre: 2999:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000401:0x1696:0x0]/ may get corrupted (rc -108)
Lustre: lustre-OST0001-osc-ffff9bc60c7ed000: Connection restored to 0@lo (at 0@lo)
Lustre: ll_ost_io00_003: service thread pid 8381 was inactive for 228.011 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_003 state:D stack:0 pid:8381 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? prepare_to_wait_exclusive_return+0x80/0x130
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
? __es_insert_extent+0x2f5/0x550 [ldiskfs]
? jbd2_journal_get_write_access+0x87/0xf0
? __ldiskfs_journal_get_write_access+0x64/0xd0 [ldiskfs]
? osd_fallocate_preallocate.isra.36+0x452/0xc30 [osd_ldiskfs]
? ldiskfs_reserve_inode_write+0xbc/0x110 [ldiskfs]
? __might_sleep+0x60/0xc0
? ldiskfs_mark_inode_dirty+0xa6/0x390 [ldiskfs]
? osd_fallocate_preallocate.isra.36+0x452/0xc30 [osd_ldiskfs]
? __ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_fallocate+0x151/0x500 [osd_ldiskfs]
? ofd_object_fallocate+0x83a/0x12a0 [ofd]
? ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
? tgt_handle_request0+0x139/0xb00 [ptlrpc]
? tgt_request_handle+0x59b/0x2010 [ptlrpc]
? ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
? ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
? kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
? ret_from_fork+0x1f/0x30
INFO: task fallocate:35392 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35392 ppid:7438 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fdf9cae870b
Code: Unable to access opcode bytes at RIP 0x7fdf9cae86e1.
RSP: 002b:00007ffcb1afc3f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdf9cae870b
RDX: 000000000000784b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001c11 R11: 0000000000000246 R12: 00007ffcb1afc618
R13: 0000000000000000 R14: 00007ffcb1afc458 R15: 000000000000784b
INFO: task chmod:35534 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:35534 ppid:7431 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fa8a10e1258
Code: Unable to access opcode bytes at RIP 0x7fa8a10e122e.
RSP: 002b:00007fff5d5d4d38 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa8a10e1258
RDX: 00000000000009b2 RSI: 0000560473779500 RDI: 00000000ffffff9c
RBP: 0000560473779470 R08: 0000000000000000 R09: 00000000000009b2
R10: 0000000000000fff R11: 0000000000000246 R12: 0000560471ee7bb4
R13: 0000560473779500 R14: 000056047377a730 R15: 0000000000008000
INFO: task fallocate:44274 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:44274 ppid:7420 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f5a0c16870b
Code: Unable to access opcode bytes at RIP 0x7f5a0c1686e1.
RSP: 002b:00007ffc5687d5f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007f5a0c16870b
RDX: 0000000000004014 RSI: 0000000000000001 RDI: 0000000000000003
RBP: 0000000000000007 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000002f97 R11: 0000000000000246 R12: 00007ffc5687d818
R13: 0000000000000001 R14: 00007ffc5687d658 R15: 0000000000004014
INFO: task chown:44403 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:44403 ppid:7448 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
? __might_sleep+0x60/0xc0
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f1821072fbe
Code: Unable to access opcode bytes at RIP 0x7f1821072f94.
RSP: 002b:00007ffffe9cdf58 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffffe9ce190 RCX: 00007f1821072fbe
RDX: 000000000000620a RSI: 00005587b3d65d20 RDI: 00000000ffffff9c
RBP: 00005587b3d66f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 00000000000041da R11: 0000000000000246 R12: 00000000ffffffff
R13: 00005587b3d66fc8 R14: 000000000000620a R15: 00000000000041da
INFO: task chown:44793 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chown state:D stack:0 pid:44793 ppid:7413 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chown_common.isra.13+0x13f/0x2c0
do_fchownat+0xbc/0x170
__x64_sys_fchownat+0x2c/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fb6de2d4fbe
Code: Unable to access opcode bytes at RIP 0x7fb6de2d4f94.
RSP: 002b:00007ffc487dff58 EFLAGS: 00000246 ORIG_RAX: 0000000000000104
RAX: ffffffffffffffda RBX: 00007ffc487e0190 RCX: 00007fb6de2d4fbe
RDX: 000000000000217f RSI: 0000558242c45d20 RDI: 00000000ffffff9c
RBP: 0000558242c46f50 R08: 0000000000000000 R09: 00000000ffffffff
R10: 0000000000006eac R11: 0000000000000246 R12: 00000000ffffffff
R13: 0000558242c46fc8 R14: 000000000000217f R15: 0000000000006eac
INFO: task fallocate:50329 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:50329 ppid:7524 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f9c5eb4f70b
Code: Unable to access opcode bytes at RIP 0x7f9c5eb4f6e1.
RSP: 002b:00007ffdd3aa7658 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f9c5eb4f70b
RDX: 00000000000037df RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000004b85 R11: 0000000000000246 R12: 00007ffdd3aa7878
R13: 0000000000000000 R14: 00007ffdd3aa76b8 R15: 00000000000037df
LustreError: 89630:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff9bc620200800: inode [0x200000402:0x196f:0x0] mdc close failed: rc = -13
Lustre: ll_ost_io00_006: service thread pid 10507 was inactive for 226.947 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_io00_006 state:D stack:0 pid:10507 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
wait_transaction_locked+0xab/0x110
? woken_wake_function+0x40/0x40
add_transaction_credits+0x244/0x460
start_this_handle+0x12a/0x850
? kmem_cache_alloc+0x188/0x450
jbd2__journal_start+0x11f/0x300
__ldiskfs_journal_start_sb+0xb0/0x1c0 [ldiskfs]
? osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
osd_trans_start+0x266/0x9e0 [osd_ldiskfs]
dt_trans_start+0x20/0x70 [obdclass]
ofd_trans_start+0x7d/0xf0 [ofd]
ofd_object_fallocate+0x6cf/0x12a0 [ofd]
ofd_fallocate_hdl+0x9f9/0xd20 [ofd]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
3[103222]: segfault at 0 ip 000055db3bfe1b47 sp 00007ffd6d52d970 error 6 in 3[55db3bfdd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[103594]: segfault at 8 ip 00007f3ee1387875 sp 00007ffcdea9d820 error 4 in ld-2.28.so[7f3ee1366000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 5504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 0000000018b85112/0xd63ccd2c3a671dff lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x1d5:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 20480->57343) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd63ccd2c3a671ddc expref: 12 pid: 19708 timeout: 579 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: operation ldlm_enqueue to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3001:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x1eba:0x0]/ may get corrupted (rc -108)
LustreError: 107280:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff9bc60c7ed000: namespace resource [0x300000400:0x1d5:0x0].0x0 (ffff9bc60e6b4100) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 107280:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection restored to 0@lo (at 0@lo)
INFO: task fallocate:35392 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:fallocate state:D stack:0 pid:35392 ppid:7438 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? wait_for_completion+0xb5/0x1f0
schedule+0xc4/0x180
schedule_timeout+0x121/0x190
? lustre_msg_size+0x5c/0x1a0 [ptlrpc]
? do_raw_spin_unlock+0x79/0x190
wait_for_completion+0xf8/0x1f0
osc_io_setattr_end+0x21a/0x310 [osc]
cl_io_end+0x5f/0x190 [obdclass]
? lov_io_iter_fini_wrapper+0x70/0x70 [lov]
lov_io_end_wrapper+0x116/0x130 [lov]
lov_io_call.isra.20+0x97/0x1c0 [lov]
lov_io_end+0xc0/0x1a0 [lov]
cl_io_end+0x5f/0x190 [obdclass]
cl_io_loop+0xfb/0x2f0 [obdclass]
ll_fallocate+0xa98/0xcc0 [lustre]
vfs_fallocate+0x175/0x440
ksys_fallocate+0x51/0xb0
__x64_sys_fallocate+0x26/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fdf9cae870b
Code: Unable to access opcode bytes at RIP 0x7fdf9cae86e1.
RSP: 002b:00007ffcb1afc3f8 EFLAGS: 00000246 ORIG_RAX: 000000000000011d
RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fdf9cae870b
RDX: 000000000000784b RSI: 0000000000000000 RDI: 0000000000000003
RBP: 0000000000000006 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000001c11 R11: 0000000000000246 R12: 00007ffcb1afc618
R13: 0000000000000000 R14: 00007ffcb1afc458 R15: 000000000000784b
INFO: task chmod:35534 blocked for more than 120 seconds.
Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:chmod state:D stack:0 pid:35534 ppid:7431 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
__mutex_lock.isra.10+0x93e/0xec0
__mutex_lock_slowpath+0x23/0x30
mutex_lock+0x5f/0x80
vvp_io_setattr_start+0x74/0x5f0 [lustre]
cl_io_start+0x6a/0x1a0 [obdclass]
cl_io_loop+0xe5/0x2f0 [obdclass]
cl_setattr_ost+0x3c2/0x530 [lustre]
ll_setattr_raw+0x12f7/0x17f0 [lustre]
? from_kgid+0x1a/0x30
? privileged_wrt_inode_uidgid+0x4d/0x70
? capable_wrt_inode_uidgid+0x5c/0x80
ll_setattr+0x77/0x240 [lustre]
notify_change+0x3f0/0x790
chmod_common+0xa6/0x230
do_fchmodat+0x75/0x100
__x64_sys_fchmodat+0x22/0x30
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fa8a10e1258
Code: Unable to access opcode bytes at RIP 0x7fa8a10e122e.
RSP: 002b:00007fff5d5d4d38 EFLAGS: 00000246 ORIG_RAX: 000000000000010c
RAX: ffffffffffffffda RBX: 0000000000000001 RCX: 00007fa8a10e1258
RDX: 00000000000009b2 RSI: 0000560473779500 RDI: 00000000ffffff9c
RBP: 0000560473779470 R08: 0000000000000000 R09: 00000000000009b2
R10: 0000000000000fff R11: 0000000000000246 R12: 0000560471ee7bb4
R13: 0000560473779500 R14: 000056047377a730 R15: 0000000000008000
8[115895]: segfault at 8 ip 00007fc6d4387875 sp 00007ffee5768e40 error 4 in ld-2.28.so[7fc6d4366000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6246:0:(osd_handler.c:2095:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 6246:0:(osd_handler.c:2095:osd_trans_dump_creds()) Skipped 104 previous similar messages
Lustre: 6246:0:(osd_handler.c:2102:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 6246:0:(osd_handler.c:2102:osd_trans_dump_creds()) Skipped 104 previous similar messages
Lustre: 6246:0:(osd_handler.c:2112:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 6246:0:(osd_handler.c:2112:osd_trans_dump_creds()) Skipped 104 previous similar messages
Lustre: 6246:0:(osd_handler.c:2119:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 6246:0:(osd_handler.c:2119:osd_trans_dump_creds()) Skipped 104 previous similar messages
Lustre: 6246:0:(osd_handler.c:2126:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6246:0:(osd_handler.c:2126:osd_trans_dump_creds()) Skipped 104 previous similar messages
Lustre: 10085:0:(osd_internal.h:1463:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10085:0:(osd_internal.h:1463:osd_trans_exec_op()) Skipped 89 previous similar messages
4[125423]: segfault at 0 ip 0000564be97a2b47 sp 00007fff92a2f320 error 6 in 4[564be979e000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 5504:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 103s: evicting client at 0@lo ns: filter-lustre-OST0002_UUID lock: 00000000a80a7e13/0xd63ccd2c3a7a0e2e lrc: 3/0,0 mode: PW/PW res: [0x300000400:0x26f:0x0].0x0 rrc: 4 type: EXT [0->18446744073709551615] (req 4096->36863) gid 0 flags: 0x60000400000020 nid: 0@lo remote: 0xd63ccd2c3a7a0d24 expref: 17 pid: 6237 timeout: 694 lvb_type: 0 lru_score: 0 lru_type: 0 lru_pins: 0
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: operation ost_punch to node 0@lo failed: rc = -107
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection to lustre-OST0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-OST0002-osc-ffff9bc60c7ed000: This client was evicted by lustre-OST0002; in progress operations using this service will fail.
Lustre: 3005:0:(llite_lib.c:4333:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.62@tcp:/lustre/fid: [0x200000402:0x2651:0x0]/ may get corrupted (rc -108)
LustreError: 128491:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-OST0002-osc-ffff9bc60c7ed000: namespace resource [0x300000400:0x2c9:0x0].0x0 (ffff9bc6310b8400) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 128491:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-OST0002-osc-ffff9bc60c7ed000: Connection restored to 0@lo (at 0@lo)

LustreError: 546659:0:(statahead.c:1728:ll_deauthorize_statahead()) ASSERTION( lli->lli_stat_pid != 0 ) failed:
LustreError: 546659:0:(statahead.c:1728:ll_deauthorize_statahead()) LBUG
CPU: 9 PID: 546659 Comm: lfs Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_deauthorize_statahead+0x431/0x570 [lustre]
ll_file_release+0x4d0/0xcb0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fa03a34f7c5

Lustre: 15858:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff91e9a9cd1500 x1877106424657152/t4294968244(0) o101->b227ae57-b043-4445-906d-f827551445b2@0@lo:252/0 lens 376/816 e 0 to 0 dl 1790148287 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
LustreError: 19991:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0x166:0x0] ACL: rc = -2
LustreError: 24726:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x14c:0x0] ACL: rc = -2
Lustre: 11669:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff91ea2250e900 x1877106435529344/t4294974484(0) o101->b227ae57-b043-4445-906d-f827551445b2@0@lo:328/0 lens 376/864 e 0 to 0 dl 1790148363 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11669:0:(mdt_recovery.c:102:mdt_req_from_lrd()) Skipped 1 previous similar message
traps: 17[33084] general protection fault ip:55bac2679184 sp:7fffcfea9aa8 error:0 in 17[55bac2674000+7000]
ODEBUG: object 000000007cecad7d is on stack 00000000c73ed7dd, but NOT annotated.
WARNING: CPU: 5 PID: 11816 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) pcspkr i2c_piix4 virtio_balloon rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 5 PID: 11816 Comm: mdt00_020 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 7e b8 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffb2ce029c7338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffb2ce029c7470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff91eb3235e5a8 RDI: ffff91eb3235e5a8
RBP: ffffb2ce029c7388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffb2ce029c7108 R12: ffffffffba2c0e68
R13: ffffffffb8f05aa0 R14: 000000000003baa0 R15: ffff91e978b5f000
FS: 0000000000000000(0000) GS:ffff91eb32340000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fabaa6bf840 CR3: 000000018d8b7000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1164/0x2560 [lod]
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
lod_declare_create+0x237/0xc30 [lod]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5f/0xfb0 [mdd]
? lod_get_default_lov_striping+0x11d/0xad0 [lod]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 0a8a2985e97d7c32 ]---
LustreError: 8631:0:(ofd_objects.c:1359:ofd_id_repair_thread_main()) lustre-OST0000: failed to repair [0x2c0000402:0x62:0x0]: rc = -2
LustreError: 44077:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f86000: inode [0x200000404:0x3e0:0x0] mdc close failed: rc = -13
LustreError: 20469:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000401:0x4:0x0] ACL: rc = -2
17[54053]: segfault at 0 ip 000055ea25040b47 sp 00007ffc4a96eda0 error 6 in 17[55ea2503c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 54382:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f80800: inode [0x280000404:0x4a8:0x0] mdc close failed: rc = -13
hrtimer: interrupt took 2004170 ns
5[58477]: segfault at 8 ip 00007fd661cbf875 sp 00007ffd761eefe0 error 4 in ld-2.28.so[7fd661c9e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[62678]: segfault at 8 ip 00007fbd643d5875 sp 00007ffdc76f71b0 error 4 in ld-2.28.so[7fbd643b4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 16524:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000400:0xf:0x0] ACL: rc = -2
7[63243]: segfault at 8 ip 00007f187b89f875 sp 00007ffd0b6d5d50 error 4 in ld-2.28.so[7f187b87e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[64809]: segfault at 8 ip 00007f340c25a875 sp 00007ffd01d1e430 error 4 in ld-2.28.so[7f340c239000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[75212]: segfault at 8 ip 00007fda52c41875 sp 00007fff373b6eb0 error 4 in ld-2.28.so[7fda52c20000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[77310]: segfault at 8 ip 00007f7979903875 sp 00007ffde6f0e590 error 4 in ld-2.28.so[7f79798e2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 7[78520] general protection fault ip:56261f3442f7 sp:7ffe4a8b84b8 error:0 in 7[56261f33e000+7000]
4[78637]: segfault at 0 ip 000055c4bcf8a200 sp 00007ffd462a2038 error 6 in 4[55c4bcf88000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3[86161]: segfault at 0 ip 000055d4c3ac5200 sp 00007ffd77e15058 error 6 in 3[55d4c3ac3000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
3[86325]: segfault at 0 ip 000056004cbe8200 sp 00007ffd1fff1688 error 6 in 3[56004cbe6000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
11[88589]: segfault at 8 ip 00007f3daddae875 sp 00007fffbec8f4a0 error 4 in ld-2.28.so[7f3dadd8d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[89151]: segfault at 8 ip 00007f925c66f875 sp 00007ffc4cd33640 error 4 in ld-2.28.so[7f925c64e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 15831:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x93e:0x0] ACL: rc = -2
14[100227]: segfault at 8 ip 00007fed7d546875 sp 00007ffdfc216fc0 error 4 in ld-2.28.so[7fed7d525000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[101442]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[101442]: Failed to get EXE, ignoring: No such process
6[107217]: segfault at 8 ip 00007fe6f3bec875 sp 00007fff0b944240 error 4 in ld-2.28.so[7fe6f3bcb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[107434]: segfault at 8 ip 00007f53a54df875 sp 00007ffc2a83af30 error 4 in ld-2.28.so[7f53a54be000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[108100]: segfault at 8 ip 00007fdd5caaa875 sp 00007ffee3e0c390 error 4 in ld-2.28.so[7fdd5ca89000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[110142]: segfault at 55d31a9a8000 ip 000055d31a9a8000 sp 00007fff3a1800a0 error 14 in 7[55d31aba8000+1000]
Code: Unable to access opcode bytes at RIP 0x55d31a9a7fd6.
traps: 16[110828] general protection fault ip:55c1c7d8e195 sp:7ffee5b9b048 error:0 in 16[55c1c7d89000+7000]
16[113526]: segfault at 8 ip 00007f8e409f3875 sp 00007ffdcb070ae0 error 4 in ld-2.28.so[7f8e409d2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[114298]: segfault at 8 ip 00007f8abefd6875 sp 00007ffcfc7330b0 error 4 in ld-2.28.so[7f8abefb5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[114148]: segfault at 8 ip 00007f353bccc875 sp 00007ffd701bca70 error 4 in ld-2.28.so[7f353bcab000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[127305]: segfault at 8 ip 00007fd40f959875 sp 00007ffe21977cb0 error 4 in ld-2.28.so[7fd40f938000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[132071]: segfault at 8 ip 00007f017b36f875 sp 00007ffd4d62fa10 error 4 in ld-2.28.so[7f017b34e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[149528]: segfault at 8 ip 00007fb48aba8875 sp 00007ffde327ee40 error 4 in ld-2.28.so[7fb48ab87000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 11698:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0xd27:0x0] ACL: rc = -2
ODEBUG: object 0000000026736bb7 is on stack 000000005da158b8, but NOT annotated.
WARNING: CPU: 1 PID: 16528 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) pcspkr i2c_piix4 virtio_balloon rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 1 PID: 16528 Comm: mdt00_048 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 7e b8 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffb2ce0ecaf428 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffb2ce0ecaf560 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff91eb3225e5a8 RDI: ffff91eb3225e5a8
RBP: ffffb2ce0ecaf478 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffb2ce0ecaf1f8 R12: ffffffffba28aa48
R13: ffffffffb8f05aa0 R14: 0000000000005680 R15: ffff91e94dbce910
FS: 0000000000000000(0000) GS:ffff91eb32240000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000055ab0380e758 CR3: 00000001a9c59000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1164/0x2560 [lod]
? _raw_spin_unlock+0x16/0x30
? do_raw_spin_unlock+0x79/0x190
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
lod_declare_xattr_set+0x27c/0x1110 [lod]
dt_declare_xattr_set+0x79/0x230 [obdclass]
mdd_create_data+0x5b5/0x930 [mdd]
mdt_mfd_open+0x146d/0x16b0 [mdt]
mdt_finish_open+0x831/0xe00 [mdt]
mdt_open_by_fid_lock+0x9de/0x1180 [mdt]
mdt_reint_open+0x943/0x3c90 [mdt]
? lu_ucred+0x29/0x40 [obdclass]
? mdt_ucred+0x1d/0x30 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace 0a8a2985e97d7c33 ]---
7[166363]: segfault at 8 ip 00007f3a39b53875 sp 00007ffe61a038f0 error 4 in ld-2.28.so[7f3a39b32000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[171956]: segfault at 8 ip 00007f698e2ab875 sp 00007ffc051bd1f0 error 4 in ld-2.28.so[7f698e28a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[188004]: segfault at 8 ip 00007fba0b54f875 sp 00007ffd8e672d60 error 4 in ld-2.28.so[7fba0b52e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[190846]: segfault at 8 ip 00007f5be3ce0875 sp 00007fff9194c060 error 4 in ld-2.28.so[7f5be3cbf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[195618]: segfault at 55cc5339000d ip 000055cc5339000d sp 00007ffce5824740 error 14 in libc-2.28.so[7f45c3a06000+1cd000]
Code: Unable to access opcode bytes at RIP 0x55cc5338ffe3.
11[203408]: segfault at 0 ip 000055732e78cb47 sp 00007ffc82cac8d0 error 6 in 11[55732e788000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[207693]: segfault at 8 ip 00007ff862b10875 sp 00007ffdc2ffc800 error 4 in ld-2.28.so[7ff862aef000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[210168]: segfault at 8 ip 00007f26bd024875 sp 00007ffc11751f10 error 4 in ld-2.28.so[7f26bd003000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[212639]: segfault at 8 ip 00007f2129371875 sp 00007ffe3c90bc00 error 4 in ld-2.28.so[7f2129350000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[214695]: segfault at 0 ip 0000564add9d8b47 sp 00007ffe791cfd00 error 6 in 11[564add9d4000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[216130]: segfault at 8 ip 00007fbd559e1875 sp 00007ffd5c739af0 error 4 in ld-2.28.so[7fbd559c0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[225207]: segfault at 8 ip 00007fccf9757875 sp 00007ffdba231240 error 4 in ld-2.28.so[7fccf9736000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[228156]: segfault at 5d ip 00005555856672e8 sp 00007ffe9be1b470 error 6 in 19[555585663000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 c4 <28> 5b 5d 41 5c 41 5d 41 5e 41 5f c3 0f 1f 40 00 4c 8b 7b 20 ba 05
15[229694]: segfault at 8 ip 00007f5cd90fc875 sp 00007ffeba9dc540 error 4 in ld-2.28.so[7f5cd90db000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[230768]: segfault at 8 ip 00007fc4d56bf875 sp 00007fff49017b70 error 4 in ld-2.28.so[7fc4d569e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[240952]: segfault at 0 ip 00005612c0b6bb47 sp 00007ffd7773a910 error 6 in 19[5612c0b67000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18[241116]: segfault at 8 ip 00007f3b3eeb7875 sp 00007fff63402480 error 4 in ld-2.28.so[7f3b3ee96000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[258748]: segfault at 8 ip 00007f197a73a875 sp 00007ffe4f337e80 error 4 in ld-2.28.so[7f197a719000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: lustre-MDT0002: trigger partial OI scrub for RPC inconsistency, checking FID [0x280000404:0x1a8e:0x0]/0xa): rc = 0
LustreError: 24726:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x1e63:0x0] ACL: rc = -2
4[280659]: segfault at 0 ip 00005653f8af4b47 sp 00007ffc6aa17630 error 6 in 4[5653f8af0000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[281678]: segfault at 8 ip 00007fd6e0446875 sp 00007ffcb0199890 error 4 in ld-2.28.so[7fd6e0425000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 18[285029] trap invalid opcode ip:55c15e14347a sp:3863f70 error:0 in 18 (deleted)[55c15e13e000+7000]
systemd-coredump[286197]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[286197]: Failed to get EXE, ignoring: No such process
traps: 1[287477] trap invalid opcode ip:55b81c8d9a22 sp:7ffc107b7e68 error:0 in 1[55b81c8d4000+7000]
6[287939]: segfault at 63 ip 000056325d2bfdd8 sp 00007ffee5c18d58 error 6 in 6[56325d2ba000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 77 61 72 65 20 46 6f 75 6e 64 61 74 69 6f 6e 2c 20 49 6e 63 2e
11[288617]: segfault at 55eef8169000 ip 000055eef8169000 sp 00007ffdb88a6e78 error 14 in 11[55eef8369000+1000]
Code: Unable to access opcode bytes at RIP 0x55eef8168fd6.
LustreError: 104827:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x1cd0:0x0] ACL: rc = -2
LustreError: 104827:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 1 previous similar message
1[293022]: segfault at 8 ip 00007fe09cac1875 sp 00007fff7a4eb500 error 4 in ld-2.28.so[7fe09caa0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 298984:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f86000: inode [0x280000404:0x20f5:0x0] mdc close failed: rc = -13
LustreError: 300403:0:(statahead.c:2479:start_statahead_thread()) lustre: cannot start ll_sa_300403 thread for 'racer2': rc = -4
1[303564]: segfault at 8 ip 00007fbe22412875 sp 00007ffe7b683f50 error 4 in ld-2.28.so[7fbe223f1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[304054]: segfault at 0 ip 0000561181d67c10 sp 00007ffe12843298 error 6 in 0[561181d66000+7000]
Code: 48 89 e5 74 0c 48 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 304336:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f86000: inode [0x200000404:0x2ca0:0x0] mdc close failed: rc = -13
19[306567]: segfault at 8 ip 00007f428018f875 sp 00007ffc1b182ee0 error 4 in ld-2.28.so[7f428016e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 3[311647] general protection fault ip:558b6e93b0d6 sp:7fff6314f8f8 error:0 in 3[558b6e936000+7000]
LustreError: 317916:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f80800: inode [0x280000404:0x22a3:0x0] mdc close failed: rc = -13
9[319213]: segfault at 8 ip 00007f4c05516875 sp 00007ffd1a2201b0 error 4 in ld-2.28.so[7f4c054f5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[322747]: segfault at 8 ip 00007fd445ef2875 sp 00007ffd4ffebc20 error 4 in ld-2.28.so[7fd445ed1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 25028:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x2fff:0x0] ACL: rc = -2
LustreError: 25028:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 1 previous similar message
7[337053]: segfault at 8 ip 00007f2eee15f875 sp 00007ffc38f7e510 error 4 in ld-2.28.so[7f2eee13e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: mdt_io00_005: service thread pid 21740 was inactive for 40.860 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:mdt_io00_005 state:I stack:0 pid:21740 ppid:2 flags:0x80004080
task:mdt_out00_006 state:I stack:0 pid:32638 ppid:2 flags:0x80004080
Call Trace:
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x54d/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x30/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_lock_internal+0x219/0x5b0 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_check_lock+0x24b/0x4c0 [mdt]
mdt_reint_rename+0x1745/0x34d0 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: mdt00_010: service thread pid 11537 was inactive for 43.738 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Lustre: Skipped 1 previous similar message
task:mdt00_010 state:I stack:0 pid:11537 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x41b/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_object_find+0x21/0x30 [obdclass]
mdt_reint_unlink+0x234/0x1a30 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_reint+0x16c/0x190 [mdt]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
19[360691]: segfault at 8 ip 00007f0138f95875 sp 00007fff7abfb040 error 4 in ld-2.28.so[7f0138f74000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[361364]: segfault at 8 ip 00007fe6b9429875 sp 00007fff0f1448c0 error 4 in ld-2.28.so[7fe6b9408000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[362060]: segfault at 0 ip 000055867c0fdb47 sp 00007fff7c530010 error 6 in 19[55867c0f9000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 361611:0:(statahead.c:2479:start_statahead_thread()) lustre: cannot start ll_sa_361611 thread for 'racer0': rc = -4
4[365842]: segfault at 8 ip 00007f0e1248b875 sp 00007ffecd9f17d0 error 4 in ld-2.28.so[7f0e1246a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: mdt_out00_004: service thread pid 11727 was inactive for 76.758 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
14[368677]: segfault at 8 ip 00007fdd9f810875 sp 00007fff645a1100 error 4 in ld-2.28.so[7fdd9f7ef000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 7433:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff91e94368f800/0x894a535e49051821 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x250b:0x0].0x0 bits 0x1/0x0 rrc: 7 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x894a535e49051780 expref: 529 pid: 32638 timeout: 1646 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 30693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1790149783 with bad export cookie 9892811195874317043
Lustre: mdt00_010: service thread pid 11537 completed after 101.118s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: lustre-MDT0001-mdc-ffff91e973f86000: operation mds_reint to node 0@lo failed: rc = -107
LustreError: 21740:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff91e942793800 x1877106689597696/t0(0) o105->lustre-MDT0001@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295
Lustre: mdt_out00_001: service thread pid 7453 completed after 91.695s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0001-mdc-ffff91e973f86000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: lustre-MDT0001-mdc-ffff91e973f86000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 11727:0:(ldlm_lockd.c:1453:ldlm_handle_enqueue()) ### lock on destroyed export 000000004c7e218f ns: mdt-lustre-MDT0001_UUID lock: ffff91e976b2bc00/0x894a535e49151943 lrc: 3/0,0 mode: PR/PR res: [0x240000404:0x2582:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x894a535e490619a7 expref: 156 pid: 11727 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: mdt_out00_004: service thread pid 11727 completed after 93.211s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 371149:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f86000: inode [0x240000403:0x51d:0x0] mdc close failed: rc = -108
Lustre: mdt_io00_005: service thread pid 21740 completed after 102.396s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
LustreError: 349256:0:(mdc_request.c:1507:mdc_read_page()) lustre-MDT0001-mdc-ffff91e973f86000: [0x240000403:0x1:0x0] lock enqueue fails: rc = -5
LustreError: 349256:0:(statahead.c:1814:is_first_dirent()) lustre: reading dir [0x240000403:0x1:0x0] at 0 stat_pid = 349086 : rc = -5
LustreError: 371149:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0001-mdc-ffff91e973f86000: namespace resource [0x240000403:0x1:0x0].0x0 (ffff91e90ef94500) refcount nonzero (3) after lock cleanup; forcing cleanup.
LustreError: 349207:0:(file.c:6243:ll_inode_revalidate_fini()) lustre: revalidate FID [0x240000404:0x2431:0x0] error: rc = -108
Lustre: mdt_out00_006: service thread pid 32638 completed after 101.903s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
Lustre: lustre-MDT0001-mdc-ffff91e973f86000: Connection restored to 0@lo (at 0@lo)
8[384160]: segfault at 8 ip 00007fdab266d875 sp 00007ffc9109cac0 error 4 in ld-2.28.so[7fdab264c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 167736:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000404:0x273a:0x0] ACL: rc = -2
LustreError: 167736:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 2 previous similar messages
LustreError: 389632:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f86000: inode [0x200000403:0x3bd3:0x0] mdc close failed: rc = -13
LustreError: 389632:0:(file.c:252:ll_close_inode_openhandle()) Skipped 51 previous similar messages
1[390044]: segfault at 8 ip 00007f6ff5ea1875 sp 00007fffc7fa7b30 error 4 in ld-2.28.so[7f6ff5e80000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 400841:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f80800: inode [0x200000404:0x3d78:0x0] mdc close failed: rc = -13
13[413435]: segfault at 8 ip 00007f4004914875 sp 00007ffc64f4dd90 error 4 in ld-2.28.so[7f40048f3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[427662]: segfault at 8 ip 00007fe11ef45875 sp 00007ffed76948b0 error 4 in ld-2.28.so[7fe11ef24000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
6[441376]: segfault at 8 ip 00007feac1b1e875 sp 00007ffd1af7ab90 error 4 in ld-2.28.so[7feac1afd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[449202]: segfault at 5600e84e1000 ip 00005600e84e1000 sp 00007ffc72a7f080 error 14 in 0[5600e86e1000+1000]
Code: Unable to access opcode bytes at RIP 0x5600e84e0fd6.
LustreError: 11757:0:(mdt_handler.c:749:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x344b:0x0] ACL: rc = -2
LustreError: 11757:0:(mdt_handler.c:749:mdt_pack_acl2body()) Skipped 1 previous similar message
traps: 8[457136] trap invalid opcode ip:5592c86b86c2 sp:7ffd1cb59390 error:0 in 8[5592c86b3000+7000]
traps: 8[457830] trap invalid opcode ip:55845c78c6c2 sp:7fffac5ffc50 error:0 in 8[55845c787000+7000]
LustreError: 14896:0:(osd_io.c:1017:osd_grow_blocksize()) object [0x340000402:0x5ec:0x0]: change block size4096 -> 1048576 error rc = -95
5[471742]: segfault at 8 ip 00007f79d5868875 sp 00007ffd0840f300 error 4 in ld-2.28.so[7f79d5847000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ptlrpc_watchdog_fire: 2 callbacks suppressed
Lustre: ll_ost_out00_00: service thread pid 22436 was inactive for 40.927 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
task:ll_ost_out00_00 state:I
Lustre: Skipped 1 previous similar message
task:ll_ost_out00_00 state:I stack:0 pid:22436 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x41b/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x30/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
task:ll_ost_out00_00 state:I
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
stack:0 pid:199475 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
stack:0 pid:29665 ppid:2 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_timeout+0xb1/0x190
? __next_timer_interrupt+0x170/0x170
? do_raw_spin_unlock+0x79/0x190
ldlm_completion_ast+0xc26/0x12b0 [ptlrpc]
? woken_wake_function+0x40/0x40
ldlm_cli_enqueue_local+0x5b4/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? snprintf+0x51/0x70
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
? mdt_object_put+0x130/0x130 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
mdt_object_pdo_lock+0x41b/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_object_pdo_lock+0x41b/0x930 [mdt]
? mdt_object_put+0x130/0x130 [mdt]
? ldlm_cli_enqueue_local+0xb70/0xb70 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
mdt_parent_lock+0x93/0x370 [mdt]
? lu_name_is_valid_len+0x65/0x80 [mdt]
mdt_getattr_name_lock+0x2693/0x32e0 [mdt]
mdt_intent_getattr+0x301/0x650 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_getattr_name_lock+0x32e0/0x32e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x30/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_msg_buf+0x1f/0x70 [ptlrpc]
mdt_batch_getattr+0xfa/0x200 [mdt]
mdt_batch+0x802/0x1e17 [mdt]
? tgt_check_lookup_req+0x30/0x2e0 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
Lustre: ll_ost_out00_00: service thread pid 8621 was inactive for 41.308 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 1 previous similar message
LustreError: 7433:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0001_UUID lock: ffff91e9b56a8600/0x894a535e4955e1e7 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x2db5:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x894a535e4955dfc5 expref: 521 pid: 14185 timeout: 2171 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 7433:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 100s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff91e9790cb600/0x894a535e4953e484 lrc: 3/0,0 mode: PR/PR res: [0x280000403:0x3186:0x0].0x0 bits 0x1/0x0 rrc: 3 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0x894a535e4953e437 expref: 699 pid: 14253 timeout: 2172 lvb_type: 0 lru_score: 0 lru_type: 0
Lustre: lustre-MDT0001-mdc-ffff91e973f80800: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
LustreError: lustre-MDT0000-mdc-ffff91e973f80800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail.
LustreError: 483358:0:(file.c:252:ll_close_inode_openhandle()) lustre-clilmv-ffff91e973f80800: inode [0x200000404:0x22fc:0x0] mdc close failed: rc = -108
LustreError: lustre-MDT0001-mdc-ffff91e973f80800: This client was evicted by lustre-MDT0001; in progress operations using this service will fail.
LustreError: 483366:0:(ldlm_resource.c:1227:ldlm_resource_complain()) lustre-MDT0001-mdc-ffff91e973f80800: namespace resource [0x280000403:0x2db5:0x0].0x0 (ffff91e992ea1100) refcount nonzero (1) after lock cleanup; forcing cleanup.
LustreError: 483366:0:(ldlm_resource.c:1227:ldlm_resource_complain()) Skipped 1 previous similar message
Lustre: lustre-MDT0001-mdc-ffff91e973f80800: Connection restored to 0@lo (at 0@lo)
INFO: task dir_create.sh:10868 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dir_create.sh state:D stack:0 pid:10868 ppid:10683 flags:0x80004080
Call Trace:
__schedule+0x351/0xcb0
? cfs_trace_unlock_tcd+0x2c/0xb0 [libcfs]
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
down_read+0xa9/0x2c0
lookup_slow+0x34/0x80
walk_component+0x264/0x4e0
link_path_walk+0x417/0x890
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0x135/0x500
do_filp_open+0x99/0x140
? ll_getattr_link+0x110/0x110 [lustre]
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbf06ea8332
Code: Unable to access opcode bytes at RIP 0x7fbf06ea8308.
RSP: 002b:00007ffe10c32520 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055f205786d00 RCX: 00007fbf06ea8332
RDX: 0000000000000241 RSI: 000055f205785700 RDI: 00000000ffffff9c
RBP: 00007ffe10c32620 R08: 0000000000000020 R09: 000055f205766012
R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000003 R14: 0000000000000001 R15: 000055f205785700
INFO: task dir_create.sh:11134 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dir_create.sh state:D stack:0 pid:11134 ppid:10684 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
? cfs_trace_unlock_tcd+0x2c/0xb0 [libcfs]
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
down_read+0xa9/0x2c0
lookup_slow+0x34/0x80
walk_component+0x264/0x4e0
link_path_walk+0x417/0x890
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0x135/0x500
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fbc4b16b332
Code: Unable to access opcode bytes at RIP 0x7fbc4b16b308.
RSP: 002b:00007ffc0ea10ff0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055f6b235f210 RCX: 00007fbc4b16b332
RDX: 0000000000000241 RSI: 000055f6b2357f60 RDI: 00000000ffffff9c
RBP: 00007ffc0ea110f0 R08: 0000000000000020 R09: 000055f6b233e012
R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000003 R14: 0000000000000001 R15: 000055f6b2357f60
INFO: task dir_create.sh:11161 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:dir_create.sh state:D stack:0 pid:11161 ppid:10684 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
? cfs_trace_unlock_tcd+0x2c/0xb0 [libcfs]
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
down_read+0xa9/0x2c0
lookup_slow+0x34/0x80
walk_component+0x264/0x4e0
link_path_walk+0x417/0x890
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0x135/0x500
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7ff3bded6332
Code: Unable to access opcode bytes at RIP 0x7ff3bded6308.
RSP: 002b:00007ffd21ec46a0 EFLAGS: 00000246 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 0000563a7ce80d30 RCX: 00007ff3bded6332
RDX: 0000000000000241 RSI: 0000563a7ce86290 RDI: 00000000ffffff9c
RBP: 00007ffd21ec47a0 R08: 0000000000000020 R09: 0000563a7ce60012
R10: 00000000000001b6 R11: 0000000000000246 R12: 0000000000000000
R13: 0000000000000003 R14: 0000000000000001 R15: 0000563a7ce86290
INFO: task ls:456651 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456651 ppid:11139 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? do_raw_spin_unlock+0x79/0x190
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fa5812ae935
Code: Unable to access opcode bytes at RIP 0x7fa5812ae90b.
RSP: 002b:00007ffd4d9b2760 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055e406b7e040 RCX: 00007fa5812ae935
RDX: 0000000000090800 RSI: 000055e406b7e070 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 000055e406b7e0b0 R09: 000055e406b75012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007fa581ba7690
R13: 0000000000000000 R14: 000055e406b7e070 R15: 000055e405da8b24
INFO: task ls:456652 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456652 ppid:11139 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? getname_flags+0x73/0x320
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f32e4632935
Code: Unable to access opcode bytes at RIP 0x7f32e463290b.
RSP: 002b:00007ffdd2e1c980 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 0000559869ad8ff0 RCX: 00007f32e4632935
RDX: 0000000000090800 RSI: 0000559869ad9020 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 0000559869ad9100 R09: 0000559869ad0012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007f32e4f2b690
R13: 0000000000000000 R14: 0000559869ad9020 R15: 0000559867e8eb24
INFO: task ls:456660 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456660 ppid:11191 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? do_raw_spin_unlock+0x79/0x190
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f56db11b935
Code: Unable to access opcode bytes at RIP 0x7f56db11b90b.
RSP: 002b:00007ffd6bb7e930 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055cc6a028040 RCX: 00007f56db11b935
RDX: 0000000000090800 RSI: 000055cc6a028070 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 000055cc6a0280b0 R09: 000055cc6a01f012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007f56dba14690
R13: 0000000000000000 R14: 000055cc6a028070 R15: 000055cc69a9ab24
INFO: task ls:456661 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456661 ppid:11191 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? do_raw_spin_unlock+0x79/0x190
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7efd88445935
Code: Unable to access opcode bytes at RIP 0x7efd8844590b.
RSP: 002b:00007ffd168510f0 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055ad5963bff0 RCX: 00007efd88445935
RDX: 0000000000090800 RSI: 000055ad5963c020 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 000055ad5963c100 R09: 000055ad59633012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007efd88d3e690
R13: 0000000000000000 R14: 000055ad5963c020 R15: 000055ad58cc1b24
INFO: task ls:456662 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456662 ppid:11191 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? getname_flags+0x73/0x320
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f7b191f2935
Code: Unable to access opcode bytes at RIP 0x7f7b191f290b.
RSP: 002b:00007ffd415bf2c0 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 0000558275fa8040 RCX: 00007f7b191f2935
RDX: 0000000000090800 RSI: 0000558275fa8070 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 0000558275fa80b0 R09: 0000558275f9f012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007f7b19aeb690
R13: 0000000000000000 R14: 0000558275fa8070 R15: 0000558274e5bb24
INFO: task ls:456663 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456663 ppid:11191 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? getname_flags+0x73/0x320
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f823094d935
Code: Unable to access opcode bytes at RIP 0x7f823094d90b.
RSP: 002b:00007ffe5a39c090 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000056053ba79040 RCX: 00007f823094d935
RDX: 0000000000090800 RSI: 000056053ba79070 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 000056053ba790b0 R09: 000056053ba70012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007f8231246690
R13: 0000000000000000 R14: 000056053ba79070 R15: 0000560539a3cb24
INFO: task ls:456664 blocked for more than 120 seconds.
Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:ls state:D stack:0 pid:456664 ppid:11191 flags:0x80000080
Call Trace:
__schedule+0x351/0xcb0
schedule+0xc4/0x180
schedule_preempt_disabled+0x25/0x40
rwsem_down_read_slowpath+0x44e/0x600
? lookup_fast+0x100/0x4d0
down_read+0xa9/0x2c0
do_last+0x9f5/0xf90
? nd_jump_root+0xe5/0x160
? path_init+0x437/0x520
path_openat+0xf7/0x500
? getname_flags+0x73/0x320
do_filp_open+0x99/0x140
? do_raw_spin_unlock+0x79/0x190
? _raw_spin_unlock+0x16/0x30
? __alloc_fd+0x12a/0x240
do_sys_openat2+0x2aa/0x410
do_sys_open+0x75/0xa0
__x64_sys_openat+0x28/0x40
do_syscall_64+0xc6/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7fd99bf18935
Code: Unable to access opcode bytes at RIP 0x7fd99bf1890b.
RSP: 002b:00007ffd3064ad50 EFLAGS: 00000287 ORIG_RAX: 0000000000000101
RAX: ffffffffffffffda RBX: 000055fdec8d7ff0 RCX: 00007fd99bf18935
RDX: 0000000000090800 RSI: 000055fdec8d8020 RDI: 00000000ffffff9c
RBP: 0000000000000000 R08: 000055fdec8d8100 R09: 000055fdec8cf012
R10: 0000000000000000 R11: 0000000000000287 R12: 00007fd99c811690
R13: 0000000000000000 R14: 000055fdec8d8020 R15: 000055fdeb277b24
Lustre: mdt00_051: service thread pid 19708 was inactive for 227.907 seconds. Watchdog stack traces are limited to 3 per 300 seconds, skipping this one.
Lustre: Skipped 4 previous similar messages
15[513138]: segfault at 8 ip 00007f7c192e9875 sp 00007ffef27bb270 error 4 in ld-2.28.so[7f7c192c8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: lustre-MDT0002-osp-MDT0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 15839:0:(ldlm_request.c:103:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1790150194, 306s ago); not entering recovery in server code, just going back to sleep ns: mdt-lustre-MDT0002_UUID lock: ffff91e943fe3600/0x894a535e4955ed7f lrc: 3/1,0 mode: --/PR res: [0x280000403:0x1:0x0].0x0 bits 0x12/0x0 rrc: 15 type: IBT flags: 0x40210400000020 pid: 15839 initiator: MDT2 lru_score: 0 lru_type: 0
LustreError: 21740:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1790150194, 306s ago), entering recovery for lustre-MDT0002_UUID@0@lo ns: lustre-MDT0002-osp-MDT0000 lock: ffff91ea51d86200/0x894a535e4955f631 lrc: 4/0,1 mode: --/CW res: [0x280000403:0x1:0x0].0x0 bits 0x2/0x0 rrc: 2 type: IBT gid 0 flags: 0x1000000000000 nid: local remote: 0x894a535e4955f638 expref: -99 pid: 21740 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: 15839:0:(ldlm_request.c:103:ldlm_expired_completion_wait()) Skipped 4 previous similar messages
Lustre: lustre-MDT0002: Received new MDS connection from 0@lo, keep former export from same NID
Lustre: lustre-MDT0002-osp-MDT0000: Connection restored to 0@lo (at 0@lo)
LustreError: 21740:0:(ldlm_request.c:123:ldlm_expired_completion_wait()) Skipped 1 previous similar message
Lustre: Skipped 1 previous similar message
LustreError: dumping log to /tmp/lustre-log.1790150500.167738
LustreError: 199475:0:(ldlm_request.c:103:ldlm_expired_completion_wait()) ### lock timed out (enqueued at 1790150209, 324s ago); not entering recovery in server code, just going back to sleep ns: mdt-lustre-MDT0000_UUID lock: ffff91e973c44800/0x894a535e4957f29f lrc: 3/1,0 mode: --/CR res: [0x200000402:0x43:0x0].0x0 bits 0x2/0x0 rrc: 13 type: IBT flags: 0x40210000000000 pid: 199475 initiator: MDT0 lru_score: 0 lru_type: 0
LustreError: 199475:0:(ldlm_request.c:103:ldlm_expired_completion_wait()) Skipped 7 previous similar messages

LustreError: 11202:0:(statahead.c:1728:ll_deauthorize_statahead()) ASSERTION( lli->lli_stat_pid != 0 ) failed:
LustreError: 11202:0:(statahead.c:1728:ll_deauthorize_statahead()) LBUG
CPU: 0 PID: 11202 Comm: lfs Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_deauthorize_statahead+0x431/0x570 [lustre]
ll_file_release+0x4d0/0xcb0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f6fae9157c5

LustreError: 743196:0:(statahead.c:1728:ll_deauthorize_statahead()) ASSERTION( lli->lli_stat_pid != 0 ) failed:
LustreError: 743196:0:(statahead.c:1728:ll_deauthorize_statahead()) LBUG
CPU: 14 PID: 743196 Comm: lfs Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_deauthorize_statahead+0x431/0x570 [lustre]
ll_file_release+0x4d0/0xcb0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f0eb892a7c5

traps: 4[13455] trap invalid opcode ip:5575f0e3a166 sp:7fff835d7af8 error:0 in 4[5575f0e34000+7000]
Lustre: 13096:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff89a2a12d6900 x1874964278140928/t4294968165(0) o101->6fa7b0dc-22f0-45c4-9de4-1603530ec129@0@lo:349/0 lens 376/864 e 0 to 0 dl 1788105354 ref 1 fl Interpret:H/602/0 rc 0/0 job:'cat.0' uid:0 gid:0 projid:0
traps: 10[14844] trap invalid opcode ip:55ace993e166 sp:7ffd51dc4118 error:0 in 9[55ace9938000+7000]
17[21486]: segfault at 8 ip 00007f08185ad875 sp 00007ffc0c651a20 error 4 in ld-2.28.so[7f081858c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[24617]: segfault at 8 ip 00007fdbc1f5a875 sp 00007ffc596418a0 error 4 in ld-2.28.so[7fdbc1f39000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[25018]: segfault at 8 ip 00007fc13eca7875 sp 00007ffca1a36d10 error 4 in ld-2.28.so[7fc13ec86000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 16079:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000403:0x7:0x0] doesn't exist!: rc = -14
10[32093]: segfault at 8 ip 00007fcb5c104875 sp 00007ffd98a6ad90 error 4 in ld-2.28.so[7fcb5c0e3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 12588:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff89a23368cd00 x1874964292338816/t4294974700(0) o101->6fa7b0dc-22f0-45c4-9de4-1603530ec129@0@lo:412/0 lens 376/864 e 0 to 0 dl 1788105417 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 11405:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff89a27db56200 x1874964292844800/t4294973834(0) o101->6fa7b0dc-22f0-45c4-9de4-1603530ec129@0@lo:415/0 lens 376/840 e 0 to 0 dl 1788105420 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
ODEBUG: object 00000000e601e358 is on stack 000000004392717b, but NOT annotated.
WARNING: CPU: 15 PID: 13256 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 15 PID: 13256 Comm: mdt00_035 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e b2 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffff94ae8e98b338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffff94ae8e98b470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff89a4325de5a8 RDI: ffff89a4325de5a8
RBP: ffff94ae8e98b388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffff94ae8e98b108 R12: ffffffffb44b7b08
R13: ffffffffb3105aa0 R14: 0000000000032740 R15: ffff89a296f95dc0
FS: 0000000000000000(0000) GS:ffff89a4325c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f3c8f459020 CR3: 000000019d9fe000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
? slab_post_alloc_hook+0x6d/0x390
? __check_object_size+0x10e/0x26d
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? do_raw_spin_unlock+0x79/0x190
lod_declare_create+0x237/0xc30 [lod]
? osp_xattr_get+0x206/0x14f0 [osp]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5a/0xed0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace ca10d5ad0a632b65 ]---
traps: 16[36526] trap invalid opcode ip:55782a17f92a sp:7fffc696a818 error:0 in 16[55782a17a000+7000]
11[40394]: segfault at 0 ip 0000561fce7df35c sp 00007ffc474ab238 error 6 in 11[561fce7da000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 4e 40 00 00 00 00 00 20 ac 40 00 00 00 00 00 18 f5 40 38 2e 33 30 <00> 43 00 77 72 69 74 65 20 65 72 72 6f 72 00 25 73 3a 20 25 73 00
16[52134]: segfault at 8 ip 00007f904f4be875 sp 00007fffca1dfe50 error 4 in ld-2.28.so[7f904f49d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11455:0:(out_handler.c:886:out_tx_end()) lustre-MDT0002-osd: error during execution of #2 from /home/green/git/lustre-release/lustre/ptlrpc/../target/out_handler.c:565: rc = -2
LustreError: 11455:0:(out_lib.c:1177:out_tx_index_delete_undo()) lustre-MDT0002-osd: Oops, can not rollback index_delete yet: rc = -524
LustreError: 15947:0:(llog_cat_server.c:492:llog_cat_add_rec()) llog_write_rec -116: lh=ffff89a293ebe400
LustreError: 15947:0:(update_trans.c:1059:top_trans_stop()) lustre-MDT0002-osp-MDT0001: write updates failed: rc = -116
LustreError: 53980:0:(llite_lib.c:2160:ll_md_setattr()) md_setattr fails: rc = -116
LustreError: 14359:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 11812:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 39566:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 39566:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 2 previous similar messages
LustreError: 34473:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 34473:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 4 previous similar messages
LustreError: 61248:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a27dee8000: inode [0x240000403:0x5ae:0x0] mdc close failed: rc = -13
LustreError: 23259:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 23259:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 43 previous similar messages
LustreError: 12755:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 12755:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 39 previous similar messages
6[67437]: segfault at 2 ip 000055622e7e43b8 sp 00007fffd3a5b3e8 error 4 in 6[55622e7de000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ff <a6> 01 00 00 00 42 0e 10 8f 02 45 0e 18 8e 03 45 0e 20 8d 04 42 0e
16[68964]: segfault at 559d53980f24 ip 0000559d53980f24 sp 00007fff6b1516b8 error 7 in 16[559d5397c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[74182]: segfault at 8 ip 00007eff023e1875 sp 00007ffeac035fe0 error 4 in ld-2.28.so[7eff023c0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[74171]: segfault at 0 ip 000055881e224b47 sp 00007fff91fb1200 error 6 in 13[55881e220000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 23143:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 23143:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 114 previous similar messages
15[75788]: segfault at 8 ip 00007f628a63e875 sp 00007ffc0f70b280 error 4 in ld-2.28.so[7f628a61d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 16[81802] trap invalid opcode ip:55aeedb2f7e6 sp:7ffee7bd5650 error:0 in 16[55aeedb2d000+7000]
LustreError: 13565:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0x687:0x0] ACL: rc = -2
traps: 18[85642] general protection fault ip:55e3dde323ec sp:7ffcb9174590 error:0 in 18[55e3dde2d000+7000]
traps: 18[85928] general protection fault ip:55745f70c3ec sp:7ffc346f1da0 error:0 in 18[55745f707000+7000]
LustreError: 15618:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 15618:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 190 previous similar messages
10[98684]: segfault at 5604b885d000 ip 00005604b885d000 sp 00007ffeff4df5d8 error 14 in 10[5604b8a5d000+1000]
Code: Unable to access opcode bytes at RIP 0x5604b885cfd6.
3[122799]: segfault at 8 ip 00007ff54dc3b875 sp 00007fff60c56370 error 4 in ld-2.28.so[7ff54dc1a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 39591:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0xf17:0x0] ACL: rc = -2
LustreError: 11743:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 11743:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 500 previous similar messages
traps: 18[133378] trap invalid opcode ip:55a3201d0d52 sp:7fff8b4fad58 error:0 in 18[55a3201ca000+7000]
10[136171]: segfault at 40 ip 0000556c002843e0 sp 00007ffc6cce9ef0 error 4 in 13[556c00280000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <4c> 8b 53 40 4c 8b 4b 38 ba 05 00 00 00 48 8d 35 0c 19 00 00 48 8b
LustreError: 141827:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a2920a0800: inode [0x240000403:0xf38:0x0] mdc close failed: rc = -13
15[148412]: segfault at 8 ip 00007f1d60555875 sp 00007ffc8d8e65e0 error 4 in ld-2.28.so[7f1d60534000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[152048]: segfault at 8 ip 00007f3d63093875 sp 00007ffccd3686f0 error 4 in ld-2.28.so[7f3d63072000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 153550:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a2920a0800: inode [0x240000404:0xf2e:0x0] mdc close failed: rc = -13
13[158989]: segfault at 5616e9626000 ip 00005616e9626000 sp 00007fff293151d8 error 14 in 13[5616e9826000+1000]
Code: Unable to access opcode bytes at RIP 0x5616e9625fd6.
LustreError: 14222:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0001: unable to read [0x240000403:0x1278:0x0] ACL: rc = -2
LustreError: 172051:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a2920a0800: inode [0x200000403:0x1656:0x0] mdc close failed: rc = -13
LustreError: 172051:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
7[178345]: segfault at 556b2d63d21a ip 0000556ae563d214 sp 00007fff54da0460 error 6 in 7[556ae5639000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 05 00 00 00 48 8b 1b 31 ff 48 8d 35 8a 18 00 00 e8 16 d4 ff ff
LustreError: 32790:0:(osd_handler.c:219:osd_trans_start()) lustre-MDT0001: can't assign tx: rc = -2
LustreError: 14112:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x19f0:0x0] ACL: rc = -2
LustreError: 91551:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 91551:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 1018 previous similar messages
5[199512]: segfault at 8 ip 00007fa636a7d875 sp 00007ffec39e1860 error 4 in ld-2.28.so[7fa636a5c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 1[210253] trap invalid opcode ip:55fc0e288a8f sp:7f25ea982778 error:0 in 1[55fc0e282000+7000]
7[210185]: segfault at 8 ip 00007fe24f553875 sp 00007fff23772a20 error 4 in ld-2.28.so[7fe24f532000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[221604]: segfault at 8 ip 00007fccb71e0875 sp 00007fffd9eab4f0 error 4 in ld-2.28.so[7fccb71bf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ODEBUG: object 00000000af68eea6 is on stack 000000003c6e1b45, but NOT annotated.
WARNING: CPU: 9 PID: 177055 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 9 PID: 177055 Comm: mdt00_113 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e b2 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffff94aea8b3b338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffff94aea8b3b470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff89a43245e5a8 RDI: ffff89a43245e5a8
RBP: ffff94aea8b3b388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffff94aea8b3b108 R12: ffffffffb44a5fa8
R13: ffffffffb3105aa0 R14: 0000000000020be0 R15: ffff89a23d894b68
FS: 0000000000000000(0000) GS:ffff89a432440000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff0b9be5a10 CR3: 00000001bf854000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? nvlist_lookup_byte_array+0x1e/0x30 [zfs]
lod_declare_create+0x237/0xc30 [lod]
? osd_xattr_get_internal+0x116/0x170 [osd_zfs]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5a/0xed0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace ca10d5ad0a632b66 ]---
5[236513]: segfault at 0 ip 0000565355908b47 sp 00007ffd084c2ac0 error 6 in 5[565355904000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[242559]: segfault at 0 ip 0000558a045c5b47 sp 00007ffd2ed1c390 error 6 in 13[558a045c1000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
hrtimer: interrupt took 6277368 ns
LustreError: 255473:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a2920a0800: inode [0x240000404:0x1a20:0x0] mdc close failed: rc = -13
6[276843]: segfault at 8 ip 00007ff210061875 sp 00007ffec6890e60 error 4 in ld-2.28.so[7ff210040000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ODEBUG: object 00000000d2fcfa16 is on stack 00000000f5ff21a0, but NOT annotated.
WARNING: CPU: 4 PID: 12767 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 4 PID: 12767 Comm: mdt00_026 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e b2 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffff94ae8be2b338 EFLAGS: 00010006
RAX: 0000000000000050 RBX: ffff94ae8be2b470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff89a43231e5a8 RDI: ffff89a43231e5a8
RBP: ffff94ae8be2b388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffff94ae8be2b108 R12: ffffffffb448ad08
R13: ffffffffb3105aa0 R14: 0000000000005940 R15: ffff89a23d8dddc0
FS: 0000000000000000(0000) GS:ffff89a432300000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f05ec1ccf70 CR3: 00000002b469c000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
? slab_post_alloc_hook+0x6d/0x390
? __check_object_size+0x10e/0x26d
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? do_raw_spin_unlock+0x79/0x190
lod_declare_create+0x237/0xc30 [lod]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5a/0xed0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace ca10d5ad0a632b67 ]---
8[293969]: segfault at 8 ip 00007f86e05de875 sp 00007ffd580b53e0 error 4 in ld-2.28.so[7f86e05bd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 293828:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a27dee8000: inode [0x280000403:0x26c2:0x0] mdc close failed: rc = -13
ODEBUG: object 0000000074238d83 is on stack 0000000079e9210b, but NOT annotated.
WARNING: CPU: 2 PID: 15548 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 2 PID: 15548 Comm: mdt00_055 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e b2 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffff94ae93eaf338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffff94ae93eaf470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff89a43229e5a8 RDI: ffff89a43229e5a8
RBP: ffff94ae93eaf388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffff94ae93eaf108 R12: ffffffffb44e2148
R13: ffffffffb3105aa0 R14: 000000000005cd80 R15: ffff89a20b583550
FS: 0000000000000000(0000) GS:ffff89a432280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000559dcd5eff44 CR3: 000000013ae7b000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
? slab_post_alloc_hook+0x6d/0x390
? kvmalloc_node+0x60/0xe0
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? nvlist_lookup_byte_array+0x1e/0x30 [zfs]
lod_declare_create+0x237/0xc30 [lod]
? osd_xattr_get_internal+0x116/0x170 [osd_zfs]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5a/0xed0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace ca10d5ad0a632b68 ]---
17[304773]: segfault at 8 ip 00007fbc93369875 sp 00007ffcce7aeef0 error 4 in ld-2.28.so[7fbc93348000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[310448]: segfault at 8 ip 00007fe37aaa5875 sp 00007fff89209a60 error 4 in ld-2.28.so[7fe37aa84000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[311639]: segfault at 4 ip 000056381d1ad9aa sp 00007ffec7365c70 error 4 in 5[56381d1aa000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 44 8b 20 48 89 c5 48 8d 43 08 <44> 8b 4b 04 ff 73 30 4c 89 ef ff 73 28 50 44 8b 03 48 8b 4c 24 28
5[311229]: segfault at 0 ip 000055d19a07ba88 sp 00007fffa5cd5720 error 6 in 5[55d19a07a000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
17[312240]: segfault at 8 ip 00007fa0fff9c875 sp 00007fffcc412fc0 error 4 in ld-2.28.so[7fa0fff7b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 16104:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x2e1c:0x0] ACL: rc = -2
LustreError: 320955:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a27dee8000: inode [0x240000403:0x2334:0x0] mdc close failed: rc = -13
LustreError: 289821:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x2abd:0x0] ACL: rc = -2
0[334558]: segfault at 8 ip 00007ff3435e4875 sp 00007ffda2393dc0 error 4 in ld-2.28.so[7ff3435c3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 34491:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000403:0x31b4:0x0] ACL: rc = -2
6[344724]: segfault at 8 ip 00007fa29b8fb875 sp 00007ffebba81930 error 4 in ld-2.28.so[7fa29b8da000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[345152]: segfault at 0 ip 000055ac5f547b47 sp 00007fff380b0cf0 error 6 in 9[55ac5f543000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[352311]: segfault at 8 ip 00007f4ca09d2875 sp 00007ffda338a470 error 4 in ld-2.28.so[7f4ca09b1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[354910]: segfault at 0 ip 000055d8e1391b47 sp 00007ffd825a6230 error 6 in 5[55d8e138d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
4[356015]: segfault at 0 ip 000055ee306a1b47 sp 00007ffc3f546ef0 error 6 in 4[55ee3069d000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 13144:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 13144:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 1810 previous similar messages
LustreError: 356653:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a2920a0800: inode [0x240000403:0x279e:0x0] mdc close failed: rc = -13
5[364233]: segfault at 8 ip 00007fca4384e875 sp 00007ffc02cc9770 error 4 in ld-2.28.so[7fca4382d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[375685]: segfault at 8 ip 00007f03595f1875 sp 00007fff41a854b0 error 4 in ld-2.28.so[7f03595d0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[376540]: segfault at 8 ip 00007efc0acd2875 sp 00007ffce0aabf00 error 4 in ld-2.28.so[7efc0acb1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[384046]: segfault at 8 ip 00007f8de56bf875 sp 00007ffc950915d0 error 4 in ld-2.28.so[7f8de569e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 390152:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a27dee8000: inode [0x240000404:0x2918:0x0] mdc close failed: rc = -13
LustreError: 390152:0:(file.c:251:ll_close_inode_openhandle()) Skipped 2 previous similar messages
15[390591]: segfault at 8 ip 00007fea7ef5d875 sp 00007ffd176b3150 error 4 in ld-2.28.so[7fea7ef3c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[392835]: segfault at 8 ip 00007fbefec49875 sp 00007ffc0cd67a00 error 4 in ld-2.28.so[7fbefec28000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[393060]: segfault at 0 ip 0000557148bfea88 sp 00007fff1b4f2a00 error 6 in 2[557148bfd000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
13[399330]: segfault at 8 ip 00007f7aff928875 sp 00007ffe91dffcb0 error 4 in ld-2.28.so[7f7aff907000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[400367]: segfault at 8 ip 00007f10610d7875 sp 00007fff660b86a0 error 4 in ld-2.28.so[7f10610b6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 12836:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x31a6:0x0] ACL: rc = -2
LustreError: 12836:0:(mdt_handler.c:748:mdt_pack_acl2body()) Skipped 1 previous similar message
LustreError: 7489:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 102s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 0000000040c82f9b/0xf0ee26372fda4de4 lrc: 3/0,0 mode: PW/PW res: [0x300000401:0x3ec:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->3145727) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0xf0ee26372fda4db3 expref: 39 pid: 191332 timeout: 1543 lvb_type: 0 lru_score: 0 lru_type: 0
LustreError: lustre-OST0001-osc-ffff89a27dee8000: operation ost_read to node 0@lo failed: rc = -107
Lustre: lustre-OST0001-osc-ffff89a27dee8000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 3419:0:(osc_request.c:1024:osc_init_grant()) lustre-OST0001-osc-ffff89a27dee8000: granted 3407872 but already consumed 13631488
LustreError: lustre-OST0001-osc-ffff89a27dee8000: This client was evicted by lustre-OST0001; in progress operations using this service will fail.
Lustre: 3433:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x200000403:0x3fc6:0x0]// may get corrupted (rc -108)
Lustre: 3435:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x240000404:0x2cf8:0x0]/ may get corrupted (rc -108)
Lustre: 3420:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x240000404:0x2c1d:0x0]/ may get corrupted (rc -108)
Lustre: 3420:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x280000403:0x3765:0x0]/ may get corrupted (rc -108)
Lustre: 3423:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x280000403:0x3718:0x0]// may get corrupted (rc -108)
Lustre: 3434:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x280000403:0x3534:0x0]/ may get corrupted (rc -108)
Lustre: 3432:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x240000403:0x2bc8:0x0]// may get corrupted (rc -108)
Lustre: 3431:0:(llite_lib.c:4342:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.92@tcp:/lustre/fid: [0x240000403:0x2f2f:0x0]/ may get corrupted (rc -108)
LustreError: 423989:0:(ldlm_resource.c:1207:ldlm_resource_complain()) lustre-OST0001-osc-ffff89a27dee8000: namespace resource [0x300000401:0x3ec:0x0].0x0 (ffff89a2d137c100) refcount nonzero (1) after lock cleanup; forcing cleanup.
Lustre: lustre-OST0001-osc-ffff89a27dee8000: Connection restored to 0@lo (at 0@lo)
14[427321]: segfault at 8 ip 00007f017f09c875 sp 00007ffc35c2c940 error 4 in ld-2.28.so[7f017f07b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[427523]: segfault at 8 ip 00007f38d40b8875 sp 00007ffc4086c4a0 error 4 in ld-2.28.so[7f38d4097000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[429698]: segfault at 55d0d2dc5000 ip 000055d0d2dc5000 sp 00007ffd42ba9e18 error 14 in 3[55d0d2fc5000+1000]
Code: Unable to access opcode bytes at RIP 0x55d0d2dc4fd6.
3[429686]: segfault at 55e0e2471000 ip 000055e0e2471000 sp 00007ffd560a1c18 error 14 in 3[55e0e2671000+1000]
Code: Unable to access opcode bytes at RIP 0x55e0e2470fd6.
LustreError: 432838:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a27dee8000: inode [0x240000404:0x2d56:0x0] mdc close failed: rc = -13
LustreError: 432838:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
3[433876]: segfault at 8 ip 00007f41e5f50875 sp 00007ffe228560f0 error 4 in ld-2.28.so[7f41e5f2f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[433430]: segfault at 8 ip 00007f9d58bd1875 sp 00007ffe19e16300 error 4 in ld-2.28.so[7f9d58bb0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[442218]: segfault at 0 ip 0000560306378b47 sp 00007ffd63573550 error 6 in 5[560306374000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
5[444497]: segfault at 8 ip 00007f5f5406b875 sp 00007ffe69c95030 error 4 in ld-2.28.so[7f5f5404a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[453025]: segfault at 0 ip 00005555bdb15a88 sp 00007ffdf4f45440 error 6 in 4[5555bdb14000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 91551:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000403:0x3b66:0x0] ACL: rc = -2
19[464595]: segfault at 0 ip 0000000000000000 sp 00007ffc9a37a548 error 14 in 19[556117af8000+7000]
Code: Unable to access opcode bytes at RIP 0xffffffffffffffd6.
0[465018]: segfault at 0 ip 000055833d378b47 sp 00007ffc41ad10b0 error 6 in 0[55833d374000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
8[469284]: segfault at 8 ip 00007faa60b00875 sp 00007ffd7098d7e0 error 4 in ld-2.28.so[7faa60adf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[471083]: segfault at 8 ip 00007efcd6b41875 sp 00007ffdfc65acf0 error 4 in ld-2.28.so[7efcd6b20000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
11[471063]: segfault at 8 ip 00007fa8e5f8a875 sp 00007fff03a8e730 error 4 in ld-2.28.so[7fa8e5f69000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[471933]: segfault at 8 ip 00007f46f429b875 sp 00007ffe1a414700 error 4 in ld-2.28.so[7f46f427a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[472527]: segfault at 8 ip 00007f46d2a5f875 sp 00007fffc4242b30 error 4 in ld-2.28.so[7f46d2a3e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[490992]: segfault at 8 ip 00007f5b23189875 sp 00007ffedc65c640 error 4 in ld-2.28.so[7f5b23168000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[496658]: segfault at 8 ip 00007ff58b9b9875 sp 00007ffdd809b7e0 error 4 in ld-2.28.so[7ff58b998000+2f000]
4[496685]: segfault at 8 ip 00007f544bd44875 sp 00007ffd69e4a820 error 4 in ld-2.28.so[7f544bd23000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[500887]: segfault at 8 ip 00007f02ab285875 sp 00007ffe651bfe20 error 4 in ld-2.28.so[7f02ab264000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[502268]: segfault at 8 ip 00007f3c29165875 sp 00007ffcd3804c60 error 4 in ld-2.28.so[7f3c29144000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[505947]: segfault at 0 ip 000055725e320b47 sp 00007ffd34df2920 error 6 in 18[55725e31c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
12[509562]: segfault at 8 ip 00007f20d098a875 sp 00007fff64e97080 error 4 in ld-2.28.so[7f20d0969000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 16964:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0000: unable to read [0x200000404:0x4754:0x0] ACL: rc = -2
LustreError: 16964:0:(mdt_handler.c:748:mdt_pack_acl2body()) Skipped 3 previous similar messages
8[513200]: segfault at 0 ip 0000555c505ccb47 sp 00007fffc8511b50 error 6 in 8[555c505c8000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 515756:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff89a27dee8000: inode [0x200000403:0x4e4b:0x0] mdc close failed: rc = -13
8[519868]: segfault at 1b30 ip 0000000000001b30 sp 00007ffcb47044e0 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
LustreError: 15571:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 15571:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 1979 previous similar messages
ODEBUG: object 00000000f561ebbc is on stack 00000000ad97f9af, but NOT annotated.
WARNING: CPU: 3 PID: 23259 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_zfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) zfs(O) spl(O) libcfs(O) virtio_balloon pcspkr i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 3 PID: 23259 Comm: mdt00_082 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e b2 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffff94ae928bb428 EFLAGS: 00010006
RAX: 0000000000000050 RBX: ffff94ae928bb560 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff89a4322de5a8 RDI: ffff89a4322de5a8
RBP: ffff94ae928bb478 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffff94ae928bb1f8 R12: ffffffffb448f028
R13: ffffffffb3105aa0 R14: 0000000000009c60 R15: ffff89a29351c488
FS: 0000000000000000(0000) GS:ffff89a4322c0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000556794150608 CR3: 00000001643aa000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
? _raw_spin_unlock+0x16/0x30
? do_raw_spin_unlock+0x79/0x190
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
lod_declare_xattr_set+0x27c/0x1100 [lod]
dt_declare_xattr_set+0x79/0x230 [obdclass]
mdd_create_data+0x5b5/0x930 [mdd]
mdt_mfd_open+0x146d/0x16b0 [mdt]
mdt_finish_open+0x831/0xe00 [mdt]
mdt_open_by_fid_lock+0x9de/0x1180 [mdt]
mdt_reint_open+0x943/0x3c90 [mdt]
? lu_ucred+0x29/0x40 [obdclass]
? mdt_ucred+0x1d/0x30 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace ca10d5ad0a632b69 ]---
1[522414]: segfault at 8 ip 00007fe85e22d875 sp 00007fff43190550 error 4 in ld-2.28.so[7fe85e20c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[523443]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[523443]: Failed to get EXE, ignoring: No such process
12[528260]: segfault at 0 ip 000055f2fd1f2b47 sp 00007ffc76ebb5c0 error 6 in 12[55f2fd1ee000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[529244]: segfault at 8 ip 00007f3d954ea875 sp 00007fff64bddfb0 error 4 in ld-2.28.so[7f3d954c9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[530838]: segfault at 0 ip 0000560fbec05b47 sp 00007ffea1afa4f0 error 6 in 2[560fbec01000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[531636]: segfault at 8 ip 00007f1101671875 sp 00007ffdf5a87210 error 4 in ld-2.28.so[7f1101650000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[534050]: segfault at 558abeac6000 ip 0000558abeac6000 sp 00007ffcb2cd9578 error 14 in 0[558abecc6000+1000]
Code: Unable to access opcode bytes at RIP 0x558abeac5fd6.
0[533748]: segfault at 563203492000 ip 0000563203492000 sp 00007fff77c56238 error 14 in 0[563203692000+1000]
Code: Unable to access opcode bytes at RIP 0x563203491fd6.
19[544859]: segfault at 8 ip 00007ff96621a875 sp 00007ffe07c8b140 error 4 in ld-2.28.so[7ff9661f9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[551120]: segfault at 8 ip 00007fdf7dfb0875 sp 00007fff4785c490 error 4 in ld-2.28.so[7fdf7df8f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 15[554094] trap invalid opcode ip:55f7d77cb6c2 sp:7ffe56a42fd8 error:0 in 15[55f7d77c6000+7000]
8[562669]: segfault at 8 ip 00007f8621664875 sp 00007ffea3068d70 error 4 in ld-2.28.so[7f8621643000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[568852]: segfault at 8 ip 00007fb66f87f875 sp 00007ffc82313c80 error 4 in ld-2.28.so[7fb66f85e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[569167]: segfault at 8 ip 00007faa7c121875 sp 00007ffeb70c1b20 error 4 in ld-2.28.so[7faa7c100000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[571292]: segfault at 8 ip 00007f2b42977875 sp 00007ffcfb776b40 error 4 in ld-2.28.so[7f2b42956000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[579721]: segfault at 8 ip 00007f1f2b787875 sp 00007ffdfa81e270 error 4 in ld-2.28.so[7f1f2b766000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 177053:0:(mdt_handler.c:748:mdt_pack_acl2body()) lustre-MDT0002: unable to read [0x280000404:0x4b11:0x0] ACL: rc = -2
LustreError: 177053:0:(mdt_handler.c:748:mdt_pack_acl2body()) Skipped 4 previous similar messages
14[606000]: segfault at 555a4b020000 ip 0000555a4b020000 sp 00007fff47a38b08 error 14 in 14[555a4b220000+1000]
Code: Unable to access opcode bytes at RIP 0x555a4b01ffd6.
4[611085]: segfault at 8 ip 00007f9c32cd0875 sp 00007ffcbfaeb4b0 error 4 in ld-2.28.so[7f9c32caf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[612693]: segfault at 8 ip 00007f02a3d96875 sp 00007ffe47de6250 error 4 in ld-2.28.so[7f02a3d75000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[619637]: segfault at 7fa743048cd0 ip 00005606d341ba98 sp 00007ffd7a46d450 error 7 in 8[5606d3416000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 0a 00 28 43 29 00 57 72 69 74 74 65 6e 20 62 79 20 25 73 2e 0a
2[620048]: segfault at 8 ip 00007ff8ef350875 sp 00007ffda8919470 error 4 in ld-2.28.so[7ff8ef32f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[621568]: segfault at 8 ip 00007f0629960875 sp 00007ffc22b5c1a0 error 4 in ld-2.28.so[7f062993f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[624629]: segfault at 55ebf9e96000 ip 000055ebf9e96000 sp 00007ffdf6856878 error 14 in 18[55ebfa096000+1000]
Code: Unable to access opcode bytes at RIP 0x55ebf9e95fd6.
5[628028]: segfault at 8 ip 00007f151a065875 sp 00007ffe7b878c10 error 4 in ld-2.28.so[7f151a044000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
3[628327]: segfault at 8 ip 00007f7896802875 sp 00007fff1244c6c0 error 4 in ld-2.28.so[7f78967e1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[629453]: segfault at 8 ip 00007f22004c7875 sp 00007ffcd6b2bd30 error 4 in ld-2.28.so[7f22004a6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[642559]: segfault at 8 ip 00007fbbf8b5d875 sp 00007fff50db3410 error 4 in ld-2.28.so[7fbbf8b3c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[647995]: segfault at 8 ip 00007f3bd65b5875 sp 00007ffee21ba610 error 4 in ld-2.28.so[7f3bd6594000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[648891]: segfault at 8 ip 00007fdd98f7e875 sp 00007ffd6e4949e0 error 4 in ld-2.28.so[7fdd98f5d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[649810]: segfault at 55742ef30f24 ip 000055742ef30f24 sp 00007ffc90e595e8 error 7 in 4[55742ef2c000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[667704]: segfault at 8 ip 00007f2d30c2f875 sp 00007ffef147be90 error 4 in ld-2.28.so[7f2d30c0e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[668582]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[668582]: Failed to get EXE, ignoring: No such process
LustreError: 668843:0:(symlink.c:95:ll_readlink_internal()) lustre: inode [0x280000404:0x539c:0x0]: symlink not NULL terminated string of length 5
LustreError: 332292:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000404:0x36e:0x0] doesn't exist!: rc = -14
LustreError: 332292:0:(mdt_open.c:1324:mdt_cross_open()) Skipped 1699 previous similar messages
traps: 12[679668] trap invalid opcode ip:55a3c3a7081e sp:7ffd03b1bac8 error:0 in 12[55a3c3a6b000+7000]
15[686555]: segfault at 8 ip 00007f7746a7b875 sp 00007ffcf8cd25d0 error 4 in ld-2.28.so[7f7746a5a000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[693862]: segfault at 8 ip 00007f460f551875 sp 00007ffcea59c590 error 4 in ld-2.28.so[7f460f530000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[696684]: segfault at 1 ip 0000555efffc3950 sp 00007ffee941df48 error 6 in 15[555efffbf000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 11[710941] trap invalid opcode ip:555e8097f9a2 sp:7fffbbc03b28 error:0 in 11[555e8097a000+7000]
traps: 11[710448] trap invalid opcode ip:55d6e43a89a2 sp:7ffdf2ce6dc8 error:0 in 11[55d6e43a3000+7000]
traps: 2[723625] general protection fault ip:558bb143e2ce sp:7ffffaa53338 error:0 in 2[558bb1439000+7000]
systemd-coredump[724660]: Failed to get EXE, ignoring: No such process
15[730290]: segfault at 8 ip 00007fd7a80e5875 sp 00007ffec68d4c40 error 4 in ld-2.28.so[7fd7a80c4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[736866]: segfault at 8 ip 00007f0e1ae09875 sp 00007fff38ad42b0 error 4 in ld-2.28.so[7f0e1ade8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[736737]: segfault at 8 ip 00007f575e321875 sp 00007ffe5660d4d0 error 4 in ld-2.28.so[7f575e300000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48

LustreError: 672870:0:(statahead.c:1728:ll_deauthorize_statahead()) ASSERTION( lli->lli_stat_pid != 0 ) failed:
LustreError: 672870:0:(statahead.c:1728:ll_deauthorize_statahead()) LBUG
CPU: 2 PID: 672870 Comm: lfs Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
Call Trace:
dump_stack+0x99/0xca
lbug_with_loc.cold.4+0xd/0x86 [libcfs]
ll_deauthorize_statahead+0x431/0x570 [lustre]
ll_file_release+0x4d0/0xcb0 [lustre]
ll_dir_release+0x124/0x140 [lustre]
__fput+0x112/0x440
____fput+0x16/0x20
task_work_run+0xd2/0x120
exit_to_usermode_loop+0x1e4/0x200
do_syscall_64+0x3de/0x3f0
entry_SYSCALL_64_after_hwframe+0x49/0xae
RIP: 0033:0x7f50ed9fd7c5

12[11722]: segfault at 8 ip 00007f68c192e875 sp 00007ffce1c821c0 error 4 in ld-2.28.so[7f68c190d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[12413]: segfault at 8 ip 00007fb6aaeb3875 sp 00007ffe06380830 error 4 in ld-2.28.so[7fb6aae92000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[12409]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[12409]: Failed to get EXE, ignoring: No such process
Lustre: 10619:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9bb3335b00 x1874751435362304/t4294967873(0) o101->3d29dcc0-ed5b-40c8-8a6b-2860af095f34@0@lo:464/0 lens 376/864 e 0 to 0 dl 1787902374 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
Lustre: 7546:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 516 < left 615, rollback = 7
Lustre: 7546:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7546:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 7546:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7546:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7546:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 6751:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff9e9c30d57000 x1874751437067136/t4294968727(0) o101->3d29dcc0-ed5b-40c8-8a6b-2860af095f34@0@lo:473/0 lens 376/816 e 0 to 0 dl 1787902383 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0
4[15791]: segfault at ffffffffffffff88 ip 000055d8727024bc sp 00007ffe91ac1ff0 error 7 in 4[55d8726ff000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 43 88 04 3e 49 83 c7 01 42 0f b6 04 3a 84 c0 75 ea e9 88 fc ff
Lustre: 15330:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 15330:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 15330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15330:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15330:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 15330:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15330:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 15330:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15330:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15330:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 15330:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15330:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message
Lustre: 10904:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 615, rollback = 7
Lustre: 10904:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10904:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10904:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10904:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10904:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10904:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
6[19389]: segfault at 8 ip 00007fc7790bd875 sp 00007ffde1ccc070 error 4 in ld-2.28.so[7fc77909c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 7544:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 7544:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
Lustre: 7544:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7544:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7544:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7544:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7544:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7544:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7544:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7544:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7544:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7544:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages
Lustre: 7546:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 7546:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages
Lustre: 7546:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7546:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 7546:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7546:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 7546:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 7546:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 7546:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7546:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 7546:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7546:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0002: opcode 7: before 515 < left 615, rollback = 7
Lustre: 15330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10904:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 6 previous similar messages
Lustre: 10904:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 10904:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 10904:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10904:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 10904:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10904:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages
Lustre: 15330:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages
8[27988]: segfault at 8 ip 00007fd6b09da875 sp 00007ffc46729ea0 error 4 in ld-2.28.so[7fd6b09b9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 7545:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 18112:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 7545:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 29 previous similar messages
Lustre: 7545:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 7545:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 7545:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 7545:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 7545:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 7545:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 7545:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 7545:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 18112:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 29 previous similar messages
Lustre: 6766:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 514 < left 520, rollback = 2
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 515 < left 520, rollback = 2
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
Lustre: 10480:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 537, rollback = 7
Lustre: 10480:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 30 previous similar messages
Lustre: 10480:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 10480:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 32 previous similar messages
Lustre: 10480:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 10480:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 34 previous similar messages
Lustre: 10480:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/537/0, punch: 0/0/0, quota 4/150/0
Lustre: 10480:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 34 previous similar messages
Lustre: 10480:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 10480:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 34 previous similar messages
Lustre: 10480:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 10480:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 34 previous similar messages
12[45355]: segfault at 7f20a9f40600 ip 000055ec262fcf0e sp 00007fff19ad07f8 error 6 in 12[55ec262f7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 b4 06 <00> 00 b4 db ff ff 08 07 00 00 a4 dc ff ff 74 07 00 00 b4 dc ff ff
LustreError: 48439:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x280000403:0x3d5:0x0] mdc close failed: rc = -13
LustreError: 52537:0:(statahead.c:2479:start_statahead_thread()) lustre: cannot start ll_sa_52537 thread for 'racer2': rc = -4
0[56745]: segfault at 8 ip 00007f0c2eea0875 sp 00007ffe99251660 error 4 in ld-2.28.so[7f0c2ee7f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 18112:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 18112:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 55 previous similar messages
Lustre: 18112:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 18112:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 55 previous similar messages
Lustre: 18112:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 18112:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 55 previous similar messages
Lustre: 18112:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 18112:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 55 previous similar messages
Lustre: 18112:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 18112:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 55 previous similar messages
Lustre: 18112:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 18112:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 55 previous similar messages
3[60613]: segfault at 8 ip 00007f7814460875 sp 00007fffb429c300 error 4 in ld-2.28.so[7f781443f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[61482]: segfault at 8 ip 00007fb016290875 sp 00007ffeb4104f40 error 4 in ld-2.28.so[7fb01626f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[61357]: segfault at 8 ip 00007f6b1958f875 sp 00007ffdc792bf20 error 4 in ld-2.28.so[7f6b1956e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[73960]: segfault at 8 ip 00007fa09c3fa875 sp 00007ffc54619e30 error 4 in ld-2.28.so[7fa09c3d9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11915:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 520, rollback = 2
LustreError: 84633:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd8a51800: inode [0x280000404:0x8bf:0x0] mdc close failed: rc = -13
Lustre: 55968:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 515 < left 615, rollback = 7
Lustre: 55968:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 61 previous similar messages
Lustre: 55968:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 55968:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 63 previous similar messages
Lustre: 55968:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 55968:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 63 previous similar messages
Lustre: 55968:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 55968:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 63 previous similar messages
Lustre: 55968:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 55968:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 63 previous similar messages
Lustre: 55968:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 55968:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 63 previous similar messages
4[98420]: segfault at 8 ip 00007faefcbd7875 sp 00007ffdf5faab50 error 4 in ld-2.28.so[7faefcbb6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 102238:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x200000403:0xc90:0x0] mdc close failed: rc = -13
LustreError: 103187:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x200000403:0xc90:0x0] mdc close failed: rc = -13
18[105606]: segfault at 0 ip 00005569b82dbb47 sp 00007ffcc77e0060 error 6 in 18[5569b82d7000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
10[105463]: segfault at 0 ip 000055708cbbfb47 sp 00007ffe23b16130 error 6 in 10[55708cbbb000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 106447:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd8a51800: inode [0x280000404:0x95f:0x0] mdc close failed: rc = -13
3[107549]: segfault at 8 ip 00007f4d90479875 sp 00007ffc1971e400 error 4 in ld-2.28.so[7f4d90458000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[114578]: segfault at 8 ip 00007fd70eea0875 sp 00007ffece7631a0 error 4 in ld-2.28.so[7fd70ee7f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[125124]: segfault at 560902a12000 ip 0000560902a12000 sp 00007ffe041be770 error 14 in 8[560902c12000+1000]
Code: Unable to access opcode bytes at RIP 0x560902a11fd6.
3[126476]: segfault at 7ffc086c7b14 ip 00007ffc086c7b14 sp 00007ffc84362e90 error 14
Code: Unable to access opcode bytes at RIP 0x7ffc086c7aea.
8[132227]: segfault at 8 ip 00007f27eb568875 sp 00007fff84712af0 error 4 in ld-2.28.so[7f27eb547000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[134891]: segfault at 8 ip 00007ff1af512875 sp 00007fff226e65e0 error 4 in ld-2.28.so[7ff1af4f1000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[137218]: segfault at 0 ip 0000559abd86cb47 sp 00007ffdc5efc9e0 error 6 in 2[559abd868000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[138846]: segfault at 0 ip 000055c460b47d50 sp 00007fff215cc978 error 6 in 0[55c460b43000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
7[140291]: segfault at 8 ip 00007fa43a855875 sp 00007ffc0eaa5190 error 4 in ld-2.28.so[7fa43a834000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 15[145938] trap invalid opcode ip:55e7574a4352 sp:7ffcbfebfba8 error:0 in 15[55e75749e000+7000]
12[149216]: segfault at 8 ip 00007fc0ba0f0875 sp 00007ffddcf172d0 error 4 in ld-2.28.so[7fc0ba0cf000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[157092]: segfault at 8 ip 00007fec6b954875 sp 00007ffde52558a0 error 4 in ld-2.28.so[7fec6b933000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[158739]: segfault at 8 ip 00007f17a38e7875 sp 00007fff6e8827a0 error 4 in ld-2.28.so[7f17a38c6000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[161268]: segfault at 8 ip 00007f8089da0875 sp 00007ffdf7be8ad0 error 4 in ld-2.28.so[7f8089d7f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[166539]: segfault at 8 ip 00007f6402e1e875 sp 00007ffd4b6c3070 error 4 in ld-2.28.so[7f6402dfd000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[169777]: segfault at 8 ip 00007f306e2e5875 sp 00007ffc40fe9860 error 4 in ld-2.28.so[7f306e2c4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[170244]: segfault at 8 ip 00007fe41d493875 sp 00007ffc2d092f40 error 4 in ld-2.28.so[7fe41d472000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
1[169905]: segfault at 1b30 ip 0000000000001b30 sp 00007ffe57af8520 error 14
Code: Unable to access opcode bytes at RIP 0x1b06.
systemd-coredump[171467]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[171467]: Failed to get EXE, ignoring: No such process
19[171995]: segfault at 1 ip 000055711c7d4950 sp 00007ffc247434c8 error 6 in 19[55711c7d0000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
2[173218]: segfault at 8 ip 00007f4e31f0d875 sp 00007fff48695f80 error 4
2[173497]: segfault at 8 ip 00007f8793a7c875 sp 00007ffd758e2950 error 4
in ld-2.28.so[7f4e31eec000+2f000]
in ld-2.28.so[7f8793a5b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 55936:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 515 < left 615, rollback = 7
Lustre: 55936:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 199 previous similar messages
Lustre: 55936:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 55936:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 199 previous similar messages
Lustre: 55936:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 55936:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 199 previous similar messages
Lustre: 55936:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 55936:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 199 previous similar messages
Lustre: 55936:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 55936:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 199 previous similar messages
Lustre: 55936:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 55936:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 199 previous similar messages
15[191262]: segfault at 0 ip 0000560295848200 sp 00007ffd44cee2a8 error 6 in 15[560295846000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 6765:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 514 < left 520, rollback = 2
Lustre: 6765:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
17[213601]: segfault at 8 ip 00007f1b425da875 sp 00007fff72be2940 error 4 in ld-2.28.so[7f1b425b9000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
2[218435]: segfault at 8 ip 00007f7a7fcb8875 sp 00007ffe964f06e0 error 4 in ld-2.28.so[7f7a7fc97000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[228235]: segfault at 8 ip 00007fc32b50d875 sp 00007ffeebbe0910 error 4 in ld-2.28.so[7fc32b4ec000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[228135]: segfault at 5653f39c3000 ip 00005653f39c3000 sp 00007ffceddb4b20 error 14 in 10[5653f3bc3000+1000]
Code: Unable to access opcode bytes at RIP 0x5653f39c2fd6.
19[254840]: segfault at 8 ip 00007f2659e68875 sp 00007ffdd0c23550 error 4 in ld-2.28.so[7f2659e47000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ODEBUG: object 0000000008164dcf is on stack 00000000ef24c881, but NOT annotated.
WARNING: CPU: 8 PID: 36211 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: loop zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) dm_flakey libcfs(O) pcspkr virtio_balloon i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 8 PID: 36211 Comm: mdt00_043 Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e 94 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffbb2b51993338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffbb2b51993470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9e9d7241e5a8 RDI: ffff9e9d7241e5a8
RBP: ffffbb2b51993388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffbb2b51993108 R12: ffffffff964bb1a8
R13: ffffffff95105aa0 R14: 0000000000035de0 R15: ffff9e9c33247528
FS: 0000000000000000(0000) GS:ffff9e9d72400000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f832a4cc260 CR3: 0000000201bfa000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
? vsnprintf+0x304/0x7f0
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? do_raw_spin_unlock+0x79/0x190
lod_declare_create+0x237/0xc30 [lod]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5a/0xed0 [mdd]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace eaa129b678c508be ]---
17[262788]: segfault at 8 ip 00007f63613bc875 sp 00007fff33122590 error 4 in ld-2.28.so[7f636139b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 263448:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x240000404:0x1bf5:0x0] mdc close failed: rc = -13
16[268336]: segfault at 8 ip 00007fed79d2f875 sp 00007ffe543d3c20 error 4 in ld-2.28.so[7fed79d0e000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
16[272185]: segfault at 8 ip 00007fd1b8e30875 sp 00007fff144bb460 error 4 in ld-2.28.so[7fd1b8e0f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[275065]: segfault at 8 ip 00007fa65882c875 sp 00007fff1d255d30 error 4 in ld-2.28.so[7fa65880b000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 514 < left 520, rollback = 2
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
13[295458]: segfault at 0 ip 000055ddbf0ecc10 sp 00007ffd41c6a378 error 6 in 13[55ddbf0eb000+7000]
Code: 48 89 e5 74 0c 48 8d 3d 6e 5f 20 00 e8 39 fc ff ff e8 64 ff ff ff 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 298315:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
16[302512]: segfault at 8 ip 00007f1b854a4875 sp 00007ffcfb54cf10 error 4 in ld-2.28.so[7f1b85483000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
ODEBUG: object 0000000058849d99 is on stack 0000000064694782, but NOT annotated.
WARNING: CPU: 2 PID: 17047 at lib/debugobjects.c:368 __debug_object_init.cold.5+0x35/0x158
Modules linked in: loop zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) crc32_generic lnet(O) dm_flakey libcfs(O) pcspkr virtio_balloon i2c_piix4 rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver ata_generic ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic
CPU: 2 PID: 17047 Comm: mdt00_030 Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014
RIP: 0010:__debug_object_init.cold.5+0x35/0x158
Code: 9e 94 48 83 05 66 63 e3 02 01 89 05 9c 6b e3 02 65 48 8b 04 25 00 dd 01 00 48 8b 50 18 e8 74 4c 97 ff 48 83 05 5e 63 e3 02 01 <0f> 0b 48 83 05 5c 63 e3 02 01 48 83 05 5c 63 e3 02 01 e9 6a ee ff
RSP: 0018:ffffbb2b58663338 EFLAGS: 00010002
RAX: 0000000000000050 RBX: ffffbb2b58663470 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff9e9d7229e5a8 RDI: ffff9e9d7229e5a8
RBP: ffffbb2b58663388 R08: 0000000000000000 R09: c0000000ffff7fff
R10: 0000000000000001 R11: ffffbb2b58663108 R12: ffffffff964c0e48
R13: ffffffff95105aa0 R14: 000000000003ba80 R15: ffff9e9b580a8410
FS: 0000000000000000(0000) GS:ffff9e9d72280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fa41bf22840 CR3: 000000019c793000 CR4: 00000000000006e0
Call Trace:
? show_regs.cold.10+0x22/0x2f
? __warn+0xd0/0x160
? __debug_object_init.cold.5+0x35/0x158
? report_bug+0x120/0x150
? __irq_work_queue_local+0xc7/0x310
? do_error_trap+0xc3/0x140
? do_invalid_op+0x4b/0x60
? __debug_object_init.cold.5+0x35/0x158
? invalid_op+0x14/0x20
? __debug_object_init.cold.5+0x35/0x158
? __debug_object_init.cold.5+0x74/0x158
? lod_comp_is_ost_used+0x90/0x90 [lod]
debug_object_init+0x26/0x30
init_timer_key+0x2d/0x120
lod_ost_alloc_qos+0x1161/0x2560 [lod]
? vsnprintf+0x304/0x7f0
lod_comp_prep_create+0x1758/0x1ae0 [lod]
lod_prepare_create+0x202/0x450 [lod]
lod_declare_striped_create+0x270/0xd40 [lod]
? lod_sub_declare_create+0xcc/0x150 [lod]
? do_raw_spin_unlock+0x79/0x190
lod_declare_create+0x237/0xc30 [lod]
? osp_xattr_get+0x206/0x14f0 [osp]
dt_declare_create+0x79/0x230 [obdclass]
mdd_declare_create_object_internal+0xb6/0x2f0 [mdd]
mdd_declare_create_object.isra.20+0x5a/0xed0 [mdd]
? lod_get_default_lov_striping+0x11d/0xad0 [lod]
mdd_declare_create+0x5a/0x680 [mdd]
mdd_create+0x71a/0x2590 [mdd]
mdt_reint_open+0x35e4/0x3c90 [mdt]
? old_init_ucred_common+0x1a1/0x830 [mdt]
mdt_reint_rec+0x144/0x2c0 [mdt]
mdt_reint_internal+0x6a0/0xdd0 [mdt]
mdt_intent_open+0x187/0x5c0 [mdt]
mdt_intent_opc.constprop.47+0x153/0xfa0 [mdt]
? mdt_intent_fixup_resent+0x2e0/0x2e0 [mdt]
mdt_intent_policy+0x151/0x680 [mdt]
ldlm_lock_enqueue+0x449/0xcf0 [ptlrpc]
? cfs_hash_rw_unlock+0x15/0x40 [obdclass]
? cfs_hash_add+0xf6/0x2a0 [obdclass]
ldlm_handle_enqueue+0xcac/0x2260 [ptlrpc]
? lustre_swab_ldlm_lock_desc+0x90/0x90 [ptlrpc]
tgt_enqueue+0xd7/0x320 [ptlrpc]
tgt_handle_request0+0x139/0xb00 [ptlrpc]
tgt_request_handle+0x59b/0x2010 [ptlrpc]
ptlrpc_server_handle_request+0x443/0x13c0 [ptlrpc]
ptlrpc_main+0xcfc/0x1410 [ptlrpc]
? ptlrpc_wait_event+0x6d0/0x6d0 [ptlrpc]
kthread+0x1de/0x210
? set_kthread_struct+0x70/0x70
ret_from_fork+0x1f/0x30
---[ end trace eaa129b678c508bf ]---
12[315417]: segfault at 8 ip 00007fac4538a875 sp 00007ffd795c3290 error 4 in ld-2.28.so[7fac45369000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 315862:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x200000403:0x2d12:0x0] mdc close failed: rc = -13
LustreError: 315862:0:(file.c:251:ll_close_inode_openhandle()) Skipped 1 previous similar message
12[320186]: segfault at 8 ip 00007fc4868f4875 sp 00007ffd3452a450 error 4 in ld-2.28.so[7fc4868d3000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[333406]: segfault at 8 ip 00007fc19e6f8875 sp 00007fffa47578b0 error 4 in ld-2.28.so[7fc19e6d7000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[334390]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[334390]: Failed to get EXE, ignoring: No such process
8[337468]: segfault at 8 ip 00007fa857c3e875 sp 00007ffe86b4cc90 error 4 in ld-2.28.so[7fa857c1d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 11820:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 515 < left 615, rollback = 7
Lustre: 11820:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 375 previous similar messages
Lustre: 11820:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 11820:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 379 previous similar messages
Lustre: 11820:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 2/15/0
Lustre: 11820:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 379 previous similar messages
Lustre: 11820:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 4/150/0
Lustre: 11820:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 379 previous similar messages
Lustre: 11820:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 11820:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 379 previous similar messages
Lustre: 11820:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 11820:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 379 previous similar messages
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 513 < left 520, rollback = 2
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message
6[357449]: segfault at 8 ip 00007f0f4858e875 sp 00007fff0c960af0 error 4 in ld-2.28.so[7f0f4856d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[357979]: segfault at 8 ip 00007f4e76391875 sp 00007fffbe5a72f0 error 4 in ld-2.28.so[7f4e76370000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 358839:0:(statahead.c:2464:start_statahead_thread()) lustre: unsupported statahead pattern 0X0.
8[363039]: segfault at 8 ip 00007f8999345875 sp 00007ffc9ec5d540 error 4 in ld-2.28.so[7f8999324000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[368470]: segfault at 8 ip 00007f42e8611875 sp 00007ffe58b1b3d0 error 4 in ld-2.28.so[7f42e85f0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 15380:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0001: opcode 2: before 515 < left 520, rollback = 2
Lustre: 15380:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages
LustreError: 376029:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd8a51800: inode [0x200000404:0x3347:0x0] mdc close failed: rc = -13
1[376779]: segfault at 8 ip 00007f7cdd939875 sp 00007ffd67d1fdd0 error 4 in ld-2.28.so[7f7cdd918000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 371409:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_367972 LIST => FNAME no wakeup.
0[384812]: segfault at 8 ip 00007fd2c3aa1875 sp 00007ffe25104410 error 4 in ld-2.28.so[7fd2c3a80000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[385622]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[385622]: Failed to get EXE, ignoring: No such process
4[394811]: segfault at 8 ip 00007fdf3cde1875 sp 00007ffc3197cd60 error 4 in ld-2.28.so[7fdf3cdc0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[395608]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[395608]: Failed to get EXE, ignoring: No such process
2[398279]: segfault at 8 ip 00007f8d0a26e875 sp 00007ffec2da9cd0 error 4 in ld-2.28.so[7f8d0a24d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[403220]: segfault at 8 ip 00007fceb58ba875 sp 00007ffd427f6910 error 4 in ld-2.28.so[7fceb5899000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
18[407848]: segfault at 8 ip 00007f626815d875 sp 00007fff94382780 error 4 in ld-2.28.so[7f626813c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[414101]: segfault at 8 ip 00007f17cd7fc875 sp 00007ffdbb7cd6c0 error 4 in ld-2.28.so[7f17cd7db000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[419010]: segfault at 8 ip 00007fc4aba70875 sp 00007ffd933850f0 error 4 in ld-2.28.so[7fc4aba4f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 7549:0:(out_handler.c:886:out_tx_end()) lustre-MDT0001-osd: error during execution of #0 from /home/green/git/lustre-release/lustre/ptlrpc/../target/out_handler.c:141: rc = -2
5[430081]: segfault at 8 ip 00007fc1a6213875 sp 00007ffed5ae7ca0 error 4 in ld-2.28.so[7fc1a61f2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 442414:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x240000403:0x316e:0x0] mdc close failed: rc = -13
7[443538]: segfault at 8 ip 00007fedc1c31875 sp 00007ffe54fdfdd0 error 4 in ld-2.28.so[7fedc1c10000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
7[445166]: segfault at 8 ip 00007fd8cee5e875 sp 00007ffde06ec490 error 4 in ld-2.28.so[7fd8cee3d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[448756]: segfault at 8 ip 00007f8753c46875 sp 00007ffd104a5ee0 error 4 in ld-2.28.so[7f8753c25000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[448877]: segfault at 8 ip 00007fe7abb7e875 sp 00007fffcc50a3f0 error 4 in ld-2.28.so[7fe7abb5d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 15380:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 513 < left 520, rollback = 2
LustreError: 445081:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_441879 LIST => FNAME no wakeup.
10[456768]: segfault at 8 ip 00007fc66aee3875 sp 00007ffe78e9cb50 error 4 in ld-2.28.so[7fc66aec2000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
10[456844]: segfault at 8 ip 00007fc9fca66875 sp 00007ffd03f35240 error 4 in ld-2.28.so[7fc9fca45000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 14[457243] general protection fault ip:55d5496ccdac sp:7ffc72d4d8b8 error:0 in 14[55d5496c6000+7000]
8[463528]: segfault at 8 ip 00007f4ea3c16875 sp 00007fff0c496d30 error 4 in ld-2.28.so[7f4ea3bf5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[467246]: segfault at 8 ip 00007f99fbae6875 sp 00007fffb859f490 error 4 in ld-2.28.so[7f99fbac5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[470809]: segfault at 8 ip 00007f57567dc875 sp 00007fff23f90140 error 4 in ld-2.28.so[7f57567bb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
systemd-coredump[471553]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[471553]: Failed to get EXE, ignoring: No such process
8[475443]: segfault at 8 ip 00007fb342af5875 sp 00007ffe8e89f060 error 4 in ld-2.28.so[7fb342ad4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2
Lustre: 6764:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
8[480742]: segfault at 8 ip 00007f0663083875 sp 00007ffd77cfa990 error 4 in ld-2.28.so[7f0663062000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[484478]: segfault at 5ba207b ip 00005561af5a92f8 sp 00007ffe075a0250 error 6 in 4[5561af5a5000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 8b 7b 20 ba 05 00 00 00 31 ff 4c 8b 73 18 48 8d 35 0a 19 00 00
traps: 1[485415] trap invalid opcode ip:56005fe4e97a sp:7ffd5192d1f8 error:0 in 1[56005fe49000+7000]
11[497135]: segfault at 0 ip 00005605ef917b47 sp 00007ffc63e3d750 error 6 in 11[5605ef913000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
17[497542]: segfault at 8 ip 00007f25c9e10875 sp 00007ffc589d5e60 error 4 in ld-2.28.so[7f25c9def000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 15330:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 615, rollback = 7
Lustre: 15330:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 415 previous similar messages
Lustre: 15330:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0
Lustre: 15330:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 426 previous similar messages
Lustre: 15330:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0
Lustre: 15330:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 426 previous similar messages
Lustre: 15330:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/615/0, punch: 0/0/0, quota 1/3/0
Lustre: 15330:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 426 previous similar messages
Lustre: 15330:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0
Lustre: 15330:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 426 previous similar messages
Lustre: 15330:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0
Lustre: 15330:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 426 previous similar messages
17[503862]: segfault at 8 ip 00007fcce2b71875 sp 00007fffda682d00 error 4 in ld-2.28.so[7fcce2b50000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[508899]: segfault at 555cd44f0000 ip 0000555cd44f0000 sp 00007ffc5b0826d8 error 14 in 4[555cd46f0000+1000]
Code: Unable to access opcode bytes at RIP 0x555cd44effd6.
4[509687]: segfault at 562db1c0b000 ip 0000562db1c0b000 sp 00007ffd9dc11aa8 error 14 in 4[562db1e0b000+1000]
Code: Unable to access opcode bytes at RIP 0x562db1c0afd6.
18[530982]: segfault at 2 ip 0000563e8c98ad68 sp 00007ffd58944968 error 6 in 18[563e8c985000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 2f 77 77 77 2e 67 6e 75 2e 6f 72 67 2f 67 65 74 68 65 6c 70 2f
LustreError: 534055:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x280000404:0x39b4:0x0] mdc close failed: rc = -13
10[546653]: segfault at 8 ip 00007fbb37f45875 sp 00007fffe18decc0 error 4 in ld-2.28.so[7fbb37f24000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[548139]: segfault at 8 ip 00007f5338b30875 sp 00007ffd8fa33620 error 4 in ld-2.28.so[7f5338b0f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[548204]: segfault at 8 ip 00007f805e8c5875 sp 00007fff7a542480 error 4 in ld-2.28.so[7f805e8a4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
5[549845]: segfault at 8 ip 00007f9d277cf875 sp 00007ffd1ddc3b90 error 4 in ld-2.28.so[7f9d277ae000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
17[556063]: segfault at 0 ip 00005573f1375213 sp 00007ffeb21f28f0 error 6 in 17[5573f1373000+7000]
Code: ff 41 ff d1 e9 3c ff ff ff e8 79 f4 ff ff 66 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 53 48 85 ff 0f 84 80 00 00 00 be 2f 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: 20960:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 512 < left 520, rollback = 2
Lustre: 20960:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
traps: 0[565836] general protection fault ip:5586f3079bff sp:7ffd290c2d68 error:0 in 0[5586f3073000+7000]
9[568201]: segfault at 8 ip 00007efdd2ff6875 sp 00007ffe2453ff80 error 4 in ld-2.28.so[7efdd2fd5000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
4[571883]: segfault at 8 ip 00007f1319111875 sp 00007ffc20b8f4e0 error 4 in ld-2.28.so[7f13190f0000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[572352]: segfault at 8 ip 00007f8429921875 sp 00007ffedc8ce000 error 4 in ld-2.28.so[7f8429900000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 16724:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000403:0x3f71:0x0] doesn't exist!: rc = -14
5[585471]: segfault at 0 ip 0000557672eda200 sp 00007ffd5c49bd58 error 6 in 5[557672ed8000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
traps: 10[595376] trap invalid opcode ip:55826225512a sp:7fe6fbb11778 error:0 in 10[55826224f000+7000]
4[595516]: segfault at 8 ip 00007f60137c9875 sp 00007fffee8081d0 error 4 in ld-2.28.so[7f60137a8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[599347]: segfault at 8 ip 00007f70a5e79875 sp 00007ffd83bce410 error 4 in ld-2.28.so[7f70a5e58000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[600328]: segfault at 8 ip 00007f4a6c725875 sp 00007ffca926faf0 error 4 in ld-2.28.so[7f4a6c704000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
15[603721]: segfault at 8 ip 00007fc71da98875 sp 00007ffe5b97f2e0 error 4 in ld-2.28.so[7fc71da77000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
traps: 7[612436] general protection fault ip:55c768b10b2c sp:7ffcabdc1d90 error:0 in 7[55c768b0b000+7000]
traps: 7[611892] general protection fault ip:564444dfbb2c sp:7fff06de8f40 error:0 in 7[564444df6000+7000]
traps: 7[612034] general protection fault ip:561ad3bb1b2c sp:7ffef8a16140 error:0 in 7[561ad3bac000+7000]
systemd-coredump[612466]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[612466]: Failed to get EXE, ignoring: No such process
systemd-coredump[612495]: Failed to get COMM, falling back to the command line: No such process
systemd-coredump[612495]: Failed to get EXE, ignoring: No such process
LustreError: 606007:0:(statahead.c:1600:ll_statahead_thread()) lustre: ll_sa_604180 LIST => FNAME no wakeup.
13[615917]: segfault at 8 ip 00007f7a77e73875 sp 00007fff9aaa41a0 error 4 in ld-2.28.so[7f7a77e52000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
0[620605]: segfault at 8 ip 00007fdc84932875 sp 00007fff443cdef0 error 4 in ld-2.28.so[7fdc84911000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[624353]: segfault at 8 ip 00007f8dfbe8e875 sp 00007ffc0a232f70 error 4 in ld-2.28.so[7f8dfbe6d000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
19[630820]: segfault at 8 ip 00007fddb7825875 sp 00007ffe68498d60 error 4 in ld-2.28.so[7fddb7804000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
9[631136]: segfault at 8 ip 00007fdfa3315875 sp 00007ffdbb38df00 error 4 in ld-2.28.so[7fdfa32f4000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 632464:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff9e9bd9264000: inode [0x280000404:0x44ab:0x0] mdc close failed: rc = -13
LustreError: 632464:0:(file.c:251:ll_close_inode_openhandle()) Skipped 2 previous similar messages
4[642300]: segfault at 8 ip 00007f33d3770875 sp 00007ffec8c44d80 error 4 in ld-2.28.so[7f33d374f000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
12[642161]: segfault at 8 ip 00007f6cc6719875 sp 00007ffc2c445c90 error 4 in ld-2.28.so[7f6cc66f8000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
LustreError: 118629:0:(mdt_open.c:1324:mdt_cross_open()) lustre-MDT0001: [0x240000403:0x4831:0x0] doesn't exist!: rc = -14
0[651531]: segfault at 8 ip 00007fbd5def4875 sp 00007ffe50b989d0 error 4 in ld-2.28.so[7fbd5ded3000+2f000]
0[652193]: segfault at 8 ip 00007fbf798dc875 sp 00007ffcf533ba00 error 4
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
in ld-2.28.so[7fbf798bb000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
13[651842]: segfault at 8 ip 00007f0022cef875 sp 00007ffe9f5bf710 error 4 in ld-2.28.so[7f0022cce000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
Lustre: 15380:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 515 < left 520, rollback = 2
Lustre: 15380:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 2 previous similar messages
19[657994]: segfault at 8 ip 00007fa90994d875 sp 00007fff342c2b10 error 4 in ld-2.28.so[7fa90992c000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
14[659035]: segfault at 0 ip 000055902b95fb47 sp 00007ffd98001f80 error 6 in 14[55902b95b000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18[659749]: segfault at 8 ip 00007fe92cc33875 sp 00007ffc9f99c220 error 4 in ld-2.28.so[7fe92cc12000+2f000]
Code: 81 39 52 e5 74 64 0f 84 99 09 00 00 48 85 c0 75 e4 48 83 3d 24 f6 20 00 00 0f 85 aa 09 00 00 49 8b 47 68 49 8b 97 68 02 00 00 <48> 8b 40 08 48 85 d2 74 23 48 8b 72 08 48 01 c6 80 3e 00 74 17 48
8[671308]: segfault at 0 ip 000055e0241694bd sp 00007ffed11596f0 error 6 in 8[55e024166000+7000]
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 68 0f b6 02 84 c0 0f 84 a6 fc ff ff 0f 1f 00 4d 39 fc 76 04 <43> 88 04 3e 49 83 c7 01 42 0f b6 04 3a 84 c0 75 ea e9 88 fc ff ff