Editing crashreport #74542

ReasonCrashing FunctionWhere to cut BacktraceReports Count
kernel BUG at __list_del_entry_validlru_gen_del_folio
release_pages
__folio_batch_release
truncate_inode_pages_range
ll_lock_cancel_bits
ll_md_blocking_ast
ldlm_cancel_callback
ldlm_cli_cancel_local
ldlm_cli_cancel
ll_md_blocking_ast
ldlm_handle_bl_callback
ldlm_bl_thread_blwi
ldlm_bl_thread_main
kthread
ret_from_fork
4

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
racer test 1: racer on clients: trevis-153vm26.trevis.whamcloud.com,trevis-153vm27 DURATION=300
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
CPU: 0 PID: 66717 Comm: ldlm_bl_05 Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:__list_del_entry_valid.cold+0x31/0x47
Code: 6c 8d 99 e8 fb 69 fe ff 0f 0b 48 c7 c7 d0 6c 8d 99 e8 ed 69 fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 90 6c 8d 99 e8 d9 69 fe ff <0f> 0b 48 89 fe 4c 89 c2 48 c7 c7 58 6c 8d 99 e8 c5 69 fe ff 0f 0b
RSP: 0018:ff3adef48665ba70 EFLAGS: 00010046
RAX: 0000000000000054 RBX: ffe6e9cf45df2a00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff21ec9cbba208c0 RDI: ff21ec9cbba208c0
RBP: ff21ec9cbffd3d80 R08: 0000000000000000 R09: ff3adef48665b930
R10: ff3adef48665b928 R11: ffffffff9a3e17c8 R12: 0000000000000002
R13: 00000000ffffffff R14: ff21ec9c808b7000 R15: ff3adef48665bb58
FS: 0000000000000000(0000) GS:ff21ec9cbba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 000056062f5f5088 CR3: 000000016701a002 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x26e/0x2df
? show_trace_log_lvl+0x26e/0x2df
? lru_gen_del_folio.constprop.0+0x12f/0x1a0
? __die_body.cold+0x8/0xd
? die+0x2b/0x50
? do_trap+0xce/0x120
? __list_del_entry_valid.cold+0x31/0x47
? do_error_trap+0x65/0x80
? __list_del_entry_valid.cold+0x31/0x47
? exc_invalid_op+0x4e/0x70
? __list_del_entry_valid.cold+0x31/0x47
? asm_exc_invalid_op+0x16/0x20
? __list_del_entry_valid.cold+0x31/0x47
lru_gen_del_folio.constprop.0+0x12f/0x1a0
release_pages+0x2dc/0x4e0
__folio_batch_release+0x1b/0x30
truncate_inode_pages_range+0x133/0x3f0
? srso_alias_return_thunk+0x5/0xfbef5
? mdc_lock_match+0xb1/0x180 [mdc]
? srso_alias_return_thunk+0x5/0xfbef5
? ll_have_md_lock+0x158/0x310 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? ldlm_resource_putref+0x30/0x2d0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
ll_lock_cancel_bits+0x444/0x780 [lustre]
ll_md_blocking_ast+0x2dd/0x460 [lustre]
ldlm_cancel_callback+0x7a/0x2b0 [ptlrpc]
ldlm_cli_cancel_local+0xa6/0x490 [ptlrpc]
ldlm_cli_cancel+0xab/0x540 [ptlrpc]
ll_md_blocking_ast+0x320/0x460 [lustre]
ldlm_handle_bl_callback+0x10a/0x3f0 [ptlrpc]
ldlm_bl_thread_blwi.constprop.0+0x7c/0x320 [ptlrpc]
ldlm_bl_thread_main+0x4fb/0x5b0 [ptlrpc]
? __pfx_autoremove_wake_function+0x10/0x10
? __pfx_ldlm_bl_thread_main+0x10/0x10 [ptlrpc]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Modules linked in: nfsv3 rpcrdma rdma_cm iw_cm ib_cm nfsd nfs_acl tls ib_core mgc(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ec(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs intel_rapl_msr intel_rapl_common kvm_amd iTCO_wdt ccp rfkill iTCO_vendor_support kvm pcspkr i2c_i801 i2c_smbus lpc_ich virtio_balloon joydev sunrpc drm dm_mod fuse ext4 mbcache jbd2 ahci crct10dif_pclmul libahci crc32_pclmul libata crc32c_intel ghash_clmulni_intel virtio_blk virtio_net net_failover failover serio_raw
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
6[63587]: segfault at 7fb6e61bcd10 ip 000055c2aaed80c0 sp 00007ffde9234208 error 7 in 6[55c2aaed5000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
0[66193]: segfault at 8 ip 00007f7ebc2ec480 sp 00007ffc7bb68ad8 error 4 in ld-linux-x86-64.so.2[7f7ebc2d0000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 66647:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9c42fcf000: inode [0x200000bec:0x28:0x0] mdc close failed: rc = -2
8[67869]: segfault at 8 ip 00007f64abbdd480 sp 00007ffe9df8aaa8 error 4 in ld-linux-x86-64.so.2[7f64abbc1000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
Lustre: dir [0x200000bed:0x3f:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 69707:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0002-mdc-ff21ec9ca91c7000: dir page locate: [0x280000be8:0x9b:0x0] at 0: rc -5
Lustre: dir [0x200000bed:0x48:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 70332:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be8:0xc2:0x0]: rc = -5
LustreError: 70332:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 70525:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9c42fcf000: inode [0x240000bed:0x68:0x0] mdc close failed: rc = -2
LustreError: 70525:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 70332:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0x50:0x0]: rc = -5
LustreError: 70332:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 73204:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be8:0xc2:0x0]: rc = -5
LustreError: 73204:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
0[72729]: segfault at 8 ip 00007f816f672480 sp 00007ffd88409cc8 error 4 in ld-linux-x86-64.so.2[7f816f656000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 72432:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9ca91c7000: inode [0x2c0000be9:0x8:0x0] mdc close failed: rc = -116
Lustre: dir [0x280000be8:0x149:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 77415:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9c42fcf000: inode [0x200000bea:0x41:0x0] mdc close failed: rc = -2
LustreError: 77415:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
6[78752]: segfault at 8 ip 00007f88440e0480 sp 00007ffe4e2b0668 error 4 in ld-linux-x86-64.so.2[7f88440c4000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 77022:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0x1e2:0x0]: rc = -5
LustreError: 77022:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 77022:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 77022:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 2 previous similar messages
Lustre: dir [0x240000bef:0xee:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 2 previous similar messages
LustreError: 76786:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0002-mdc-ff21ec9ca91c7000: dir page locate: [0x280000be8:0xa6:0x0] at 0: rc -5
LustreError: 76786:0:(mdc_request.c:1510:mdc_read_page()) Skipped 1 previous similar message
LustreError: 79614:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0001-mdc-ff21ec9c42fcf000: dir page locate: [0x240000bee:0x13b:0x0] at 0: rc -5
LustreError: 79614:0:(mdc_request.c:1510:mdc_read_page()) Skipped 8 previous similar messages
LustreError: 80269:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9ca91c7000: inode [0x2c0000bec:0x1c4:0x0] mdc close failed: rc = -2
LustreError: 80269:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 81721:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be9:0x103:0x0]: rc = -5
LustreError: 81721:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 3 previous similar messages
LustreError: 81721:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 81721:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 3 previous similar messages
3[82624]: segfault at 0 ip 00005614355a6a00 sp 00007ffee931a710 error 6 in 3[5614355a6000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
16[85596]: segfault at 8 ip 00007fe2b5dc2d3e sp 00007ffd68375a50 error 4 in ld-linux-x86-64.so.2[7fe2b5da5000+27000] likely on CPU 1 (core 0, socket 1)
Code: 80 e0 00 00 00 48 85 c0 74 0b 48 83 78 08 07 0f 85 8c 2e 00 00 48 8b 85 88 fd ff ff 48 83 78 78 00 74 12 48 8b 80 88 00 00 00 <48> 83 78 08 18 0f 85 8b 2e 00 00 48 8b 9d 88 fd ff ff 48 8b 83 30
LustreError: 85193:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9ca91c7000: inode [0x2c0000bea:0x122:0x0] mdc close failed: rc = -2
LustreError: 85193:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 86393:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be7:0x1c0:0x0]: rc = -5
LustreError: 86393:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 5 previous similar messages
LustreError: 86393:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 86393:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 5 previous similar messages
LustreError: 84397:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0003-mdc-ff21ec9ca91c7000: dir page locate: [0x2c0000bd1:0x4b:0x0] at 0: rc -5
Lustre: dir [0x240000bed:0x26c:0x0] stripe 2 readdir failed: -5, directory is partially accessed!
LustreError: 84397:0:(mdc_request.c:1510:mdc_read_page()) Skipped 3 previous similar messages
Lustre: Skipped 15 previous similar messages
7[89072]: segfault at 0 ip 000055ff2efa1a00 sp 00007ffcc5fea3d0 error 6 in 7[55ff2efa1000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
9[89960]: segfault at 1c ip 0000562f06eb1b90 sp 00007ffc3eb39280 error 6 in 9[562f06eb1000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 85280:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff21ec9ca91c7000: cannot apply new layout on [0x240000bed:0x184:0x0] : rc = -5
LustreError: 85280:0:(vvp_io.c:2037:vvp_io_init()) lustre: refresh file layout [0x240000bed:0x184:0x0] error -5.
10[92958]: segfault at 8 ip 00007fa8336c7480 sp 00007ffd037bf368 error 4 in ld-linux-x86-64.so.2[7fa8336ab000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 89972:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff21ec9ca91c7000: cannot apply new layout on [0x280000be7:0x42b:0x0] : rc = -5
LustreError: 89972:0:(lov_object.c:1375:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 89972:0:(vvp_io.c:2037:vvp_io_init()) lustre: refresh file layout [0x280000be7:0x42b:0x0] error -5.
LustreError: 94407:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000bee:0x2e4:0x0]: rc = -5
LustreError: 94407:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 15 previous similar messages
LustreError: 95406:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 95406:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 15 previous similar messages
Lustre: dir [0x2c0000beb:0x3f0:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 110 previous similar messages
LustreError: 96151:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff21ec9c42fcf000: inode [0x280000be5:0x399:0x0] mdc close failed: rc = -2
LustreError: 96151:0:(file.c:248:ll_close_inode_openhandle()) Skipped 4 previous similar messages
LustreError: 97803:0:(llite_lib.c:1902:ll_update_lsm_md()) lustre: [0x240000bef:0x449:0x0] dir layout mismatch:
LustreError: 97803:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=34 count=2 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 97803:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x240000bd1:0x78:0x0]
LustreError: 97803:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
list_del corruption. prev->next should be ffe6e9cf45df2a08, but was ffe6e9cf45b67ec8
------------[ cut here ]------------
Link to test
racer test 1: racer on clients: onyx-76vm1.onyx.whamcloud.com,onyx-76vm2 DURATION=300
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP PTI
CPU: 0 PID: 128439 Comm: ldlm_bl_01 Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:__list_del_entry_valid.cold+0x31/0x47
Code: 6c 4d bc e8 fb 69 fe ff 0f 0b 48 c7 c7 d0 6c 4d bc e8 ed 69 fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 90 6c 4d bc e8 d9 69 fe ff <0f> 0b 48 89 fe 4c 89 c2 48 c7 c7 58 6c 4d bc e8 c5 69 fe ff 0f 0b
RSP: 0018:ffffbaef03f9fa70 EFLAGS: 00010046
RAX: 0000000000000054 RBX: ffffe89f056222c0 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff96f13bc208c0 RDI: ffff96f13bc208c0
RBP: ffff96f13ffd3d80 R08: 0000000000000000 R09: ffffbaef03f9f930
R10: ffffbaef03f9f928 R11: ffffffffbcfe17c8 R12: 0000000000000002
R13: 00000000ffffffff R14: ffff96f0c2b2e000 R15: ffffbaef03f9fb58
FS: 0000000000000000(0000) GS:ffff96f13bc00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00005604e98099dc CR3: 000000010bc2e002 CR4: 0000000000172ef0
Call Trace:
<TASK>
? show_trace_log_lvl+0x1c4/0x2df
? show_trace_log_lvl+0x1c4/0x2df
? lru_gen_del_folio.constprop.0+0x12f/0x1a0
? __die_body.cold+0x8/0xd
? die+0x2b/0x50
? do_trap+0xce/0x120
? __list_del_entry_valid.cold+0x31/0x47
? do_error_trap+0x65/0x80
? __list_del_entry_valid.cold+0x31/0x47
? exc_invalid_op+0x4e/0x70
? __list_del_entry_valid.cold+0x31/0x47
? asm_exc_invalid_op+0x16/0x20
? __list_del_entry_valid.cold+0x31/0x47
lru_gen_del_folio.constprop.0+0x12f/0x1a0
release_pages+0x2dc/0x4e0
? folio_batch_move_lru+0xd3/0x150
__folio_batch_release+0x1b/0x30
truncate_inode_pages_range+0x133/0x3f0
? mdc_lock_match+0xb1/0x180 [mdc]
? ll_have_md_lock+0x15b/0x310 [lustre]
? ldlm_resource_putref+0x30/0x2d0 [ptlrpc]
ll_lock_cancel_bits+0x444/0x780 [lustre]
ll_md_blocking_ast+0x2dd/0x460 [lustre]
ldlm_cancel_callback+0x7d/0x2b0 [ptlrpc]
ldlm_cli_cancel_local+0xa6/0x490 [ptlrpc]
ldlm_cli_cancel+0xab/0x540 [ptlrpc]
ll_md_blocking_ast+0x320/0x460 [lustre]
ldlm_handle_bl_callback+0x10d/0x3f0 [ptlrpc]
ldlm_bl_thread_blwi.constprop.0+0x7c/0x320 [ptlrpc]
ldlm_bl_thread_main+0x4fb/0x5b0 [ptlrpc]
? __pfx_autoremove_wake_function+0x10/0x10
? __pfx_ldlm_bl_thread_main+0x10/0x10 [ptlrpc]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Modules linked in: nfsv3 rpcrdma rdma_cm iw_cm ib_cm nfsd nfs_acl tls ib_core lustre(OE) mgc(OE) mdc(OE) lov(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) osc(OE) ksocklnd(OE) ptlrpc(OE) obdecho(OE) obdclass(OE) lnet(OE) libcfs(OE) ec(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs intel_rapl_msr intel_rapl_common kvm_intel kvm rapl iTCO_wdt iTCO_vendor_support rfkill pcspkr lpc_ich virtio_balloon i2c_i801 i2c_smbus joydev sunrpc fuse drm dm_mod ext4 mbcache jbd2 ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net virtio_blk net_failover failover serio_raw [last unloaded: libcfs(OE)]
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
LustreError: 136421:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f0c1b38000: inode [0x280000be7:0x3:0x0] mdc close failed: rc = -2
LustreError: 138464:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ffff96f0c1b38000: cannot apply new layout on [0x200000bea:0x6a:0x0] : rc = -5
LustreError: 138464:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x200000bea:0x6a:0x0] error -5.
LustreError: 141174:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be8:0xc1:0x0]: rc = -5
LustreError: 141174:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 141284:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0x80:0x0]: rc = -5
LustreError: 141284:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 142327:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0x80:0x0]: rc = -5
LustreError: 142327:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 142327:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 142327:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 140445:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f0c1b38000: inode [0x2c0000be9:0xc:0x0] mdc close failed: rc = -2
LustreError: 142588:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f12e119000: inode [0x2c0000bea:0x14c:0x0] mdc close failed: rc = -2
LustreError: 143751:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be9:0x259:0x0]: rc = -5
LustreError: 143751:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 143751:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 143751:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 2 previous similar messages
Lustre: dir [0x240000bee:0xd9:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
17[144482]: segfault at 0 ip 0000562b7382da00 sp 00007ffed15c6940 error 6 in 17[562b7382d000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 144454:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0000-mdc-ffff96f12e119000: dir page locate: [0x200000bd0:0x3a:0x0] at 0: rc -5
Lustre: dir [0x240000bec:0xb1:0x0] stripe 4 readdir failed: -5, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 147578:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be6:0xef:0x0]: rc = -5
LustreError: 147578:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 147578:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 147578:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 145776:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f12e119000: inode [0x240000bec:0x6:0x0] mdc close failed: rc = -2
LustreError: 145776:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 150281:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f12e119000: inode [0x2c0000beb:0x197:0x0] mdc close failed: rc = -2
LustreError: 150281:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
LustreError: 151903:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be8:0x26d:0x0]: rc = -5
LustreError: 151903:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 20 previous similar messages
LustreError: 151903:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 151903:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 20 previous similar messages
Lustre: dir [0x280000be8:0xa3:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 40 previous similar messages
Lustre: dir [0x280000be6:0x420:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
LustreError: 158629:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0000-mdc-ffff96f12e119000: dir page locate: [0x200000bea:0x157:0x0] at 0: rc -5
LustreError: 158629:0:(mdc_request.c:1511:mdc_read_page()) Skipped 17 previous similar messages
LustreError: 161140:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bec:0x31d:0x0]: rc = -5
LustreError: 161140:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 80 previous similar messages
LustreError: 161140:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 161140:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 80 previous similar messages
LustreError: 160532:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f0c1b38000: inode [0x240000bee:0xd9:0x0] mdc close failed: rc = -2
LustreError: 160532:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
17[166147]: segfault at 8 ip 00007f0096cde480 sp 00007ffd3deddf38 error 4 in ld-linux-x86-64.so.2[7f0096cc2000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 166906:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ffff96f0c1b38000: cannot apply new layout on [0x240000bec:0x44c:0x0] : rc = -5
LustreError: 166906:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x240000bec:0x44c:0x0] error -5.
LustreError: 169682:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f12e119000: inode [0x240000bef:0x3bc:0x0] mdc close failed: rc = -2
LustreError: 169682:0:(file.c:248:ll_close_inode_openhandle()) Skipped 7 previous similar messages
Lustre: dir [0x2c0000beb:0x4ec:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 171920:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0003-mdc-ffff96f12e119000: dir page locate: [0x2c0000bd1:0x7e:0x0] at 0: rc -5
LustreError: 176368:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000beb:0x532:0x0]: rc = -5
LustreError: 176368:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 24 previous similar messages
LustreError: 176368:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 176368:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 24 previous similar messages
3[176807]: segfault at 0 ip 000055b447027a00 sp 00007ffd09f5e0b0 error 6 in 3[55b447027000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 181968:0:(llite_nfs.c:395:ll_dir_get_parent_fid()) lustre: failure inode [0x240000bec:0x5cb:0x0] get parent: rc = -2
LustreError: 183107:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ffff96f12e119000: cannot apply new layout on [0x240000bed:0x788:0x0] : rc = -5
LustreError: 183107:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x240000bed:0x788:0x0] error -5.
3[186592]: segfault at 8 ip 00007f5e81c23480 sp 00007ffdbab1a4d8 error 4 in ld-linux-x86-64.so.2[7f5e81c07000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 187159:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f0c1b38000: inode [0x240000bef:0x5f8:0x0] mdc close failed: rc = -2
LustreError: 187159:0:(file.c:248:ll_close_inode_openhandle()) Skipped 9 previous similar messages
Lustre: dir [0x240000bef:0x80f:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 8 previous similar messages
LustreError: 187779:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0000-mdc-ffff96f12e119000: dir page locate: [0x200000bd0:0x74:0x0] at 0: rc -5
LustreError: 187779:0:(mdc_request.c:1511:mdc_read_page()) Skipped 5 previous similar messages
LustreError: 188754:0:(llite_nfs.c:395:ll_dir_get_parent_fid()) lustre: failure inode [0x240000bed:0x673:0x0] get parent: rc = -116
LustreError: 189177:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ffff96f12e119000: cannot apply new layout on [0x280000be9:0x673:0x0] : rc = -5
LustreError: 189177:0:(lov_object.c:1351:lov_layout_change()) Skipped 1 previous similar message
LustreError: 189177:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x280000be9:0x673:0x0] error -5.
LustreError: 189177:0:(vvp_io.c:1915:vvp_io_init()) Skipped 1 previous similar message
LustreError: 189880:0:(llite_lib.c:1890:ll_update_lsm_md()) lustre: [0x2c0000beb:0x70f:0x0] dir layout mismatch:
LustreError: 189880:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=3 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 189880:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x2c0000bd1:0x9b:0x0]
LustreError: 189880:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=7 index=3 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=crush:2000003 pool=
1[197221]: segfault at 8 ip 00007f08120d9480 sp 00007ffc27dd6b78 error 4 in ld-linux-x86-64.so.2[7f08120bd000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 198575:0:(llite_nfs.c:395:ll_dir_get_parent_fid()) lustre: failure inode [0x240000bec:0x6f2:0x0] get parent: rc = -116
LustreError: 198575:0:(llite_nfs.c:395:ll_dir_get_parent_fid()) Skipped 19 previous similar messages
Lustre: dir [0x240000bef:0xa0e:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 29 previous similar messages
LustreError: 200030:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0002-mdc-ffff96f12e119000: dir page locate: [0x280000bd1:0x6a:0x0] at 0: rc -5
LustreError: 200030:0:(mdc_request.c:1511:mdc_read_page()) Skipped 17 previous similar messages
LustreError: 199750:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ffff96f12e119000: cannot apply new layout on [0x280000be8:0x6ed:0x0] : rc = -5
LustreError: 199750:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x280000be8:0x6ed:0x0] error -5.
17[200582]: segfault at 0 ip 00005555b432ea00 sp 00007ffc98d7de80 error 6 in 17[5555b432e000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 111997:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep 13 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 204384:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ffff96f0c1b38000: cannot apply new layout on [0x200000bed:0xb41:0x0] : rc = -5
LustreError: 204384:0:(lov_object.c:1351:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 204384:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x200000bed:0xb41:0x0] error -5.
LustreError: 204384:0:(vvp_io.c:1915:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 203145:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0001-mdc-ffff96f0c1b38000: dir page locate: [0x240000bef:0x7cd:0x0] at 0: rc -5
LustreError: 203145:0:(mdc_request.c:1511:mdc_read_page()) Skipped 22 previous similar messages
LustreError: 171681:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep 2 [0x0:0x0:0x0] inode@0000000000000000: rc = -1
Autotest: Test running for 165 minutes (lustre-reviews_review-dne-part-9_121747.59)
LustreError: 208277:0:(llite_lib.c:1890:ll_update_lsm_md()) lustre: [0x280000be6:0xcb5:0x0] dir layout mismatch:
LustreError: 208277:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=4 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 208277:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x280000bd1:0x98:0x0]
LustreError: 208277:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 9 previous similar messages
LustreError: 208277:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=crush:2000003 pool=
LustreError: 208271:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=4 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 208271:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=crush:2000003 pool=
8[210155]: segfault at 7fcf381bcd10 ip 0000556e306bb0c0 sp 00007ffdd8593118 error 7 in 8[556e306b8000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 211641:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be8:0x953:0x0]: rc = -5
LustreError: 211641:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 263 previous similar messages
LustreError: 211641:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 211641:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 263 previous similar messages
19[212445]: segfault at 0 ip 0000563ac1c6d320 sp 00007ffd809efab8 error 6 in 19[563ac1c6c000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
18[219380]: segfault at 8 ip 00007f929263d480 sp 00007ffd6ae17e38 error 4 in ld-linux-x86-64.so.2[7f9292621000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
Lustre: dir [0x2c0000be8:0xb9a:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 40 previous similar messages
LustreError: 217485:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0003-mdc-ffff96f12e119000: dir page locate: [0x2c0000bd1:0xc2:0x0] at 0: rc -5
LustreError: 217485:0:(mdc_request.c:1511:mdc_read_page()) Skipped 1 previous similar message
5[222181]: segfault at 8 ip 00007fcdfa713480 sp 00007fff28e01c08 error 4 in ld-linux-x86-64.so.2[7fcdfa6f7000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 222489:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ffff96f12e119000: inode [0x240000bef:0xcf5:0x0] mdc close failed: rc = -2
LustreError: 222489:0:(file.c:248:ll_close_inode_openhandle()) Skipped 21 previous similar messages
LustreError: 222619:0:(llite_lib.c:1890:ll_update_lsm_md()) lustre: [0x200000bec:0xcb4:0x0] dir layout mismatch:
LustreError: 222619:0:(llite_lib.c:1890:ll_update_lsm_md()) Skipped 1 previous similar message
LustreError: 222619:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 222619:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x200000bd0:0x99:0x0]
LustreError: 222619:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 15 previous similar messages
LustreError: 222619:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
LustreError: 219868:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=2 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 219868:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=3 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=2 migrate_hash=fnv_1a_64:2 pool=
list_del corruption. prev->next should be ffffe89f056222c8, but was ffffe89f0588edc8
------------[ cut here ]------------
Link to test
racer test 1: racer on clients: onyx-156vm5.onyx.whamcloud.com,onyx-156vm6 DURATION=300
kernel BUG at lib/list_debug.c:54!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 PID: 128803 Comm: ldlm_bl_01 Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
RIP: 0010:__list_del_entry_valid.cold+0x1d/0x47
Code: c7 c7 90 6b ed 90 e8 0c 6a fe ff 0f 0b 48 89 fe 48 c7 c7 20 6c ed 90 e8 fb 69 fe ff 0f 0b 48 c7 c7 d0 6c ed 90 e8 ed 69 fe ff <0f> 0b 48 89 f2 48 89 fe 48 c7 c7 90 6c ed 90 e8 d9 69 fe ff 0f 0b
RSP: 0018:ff3ebe1743f43a70 EFLAGS: 00010046
RAX: 0000000000000054 RBX: ffacaabfc538e340 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff15d5843bb208c0 RDI: ff15d5843bb208c0
RBP: ff15d5843ffd5d80 R08: 0000000000000000 R09: ff3ebe1743f43930
R10: ff3ebe1743f43928 R11: ffffffff919e17c8 R12: 0000000000000002
R13: 00000000ffffffff R14: ff15d583fbe9e000 R15: ff3ebe1743f43b58
FS: 0000000000000000(0000) GS:ff15d5843bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fe6152c60e8 CR3: 000000012f5c8001 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x26e/0x2df
? show_trace_log_lvl+0x26e/0x2df
? lru_gen_del_folio.constprop.0+0x12f/0x1a0
? __die_body.cold+0x8/0xd
? die+0x2b/0x50
? do_trap+0xce/0x120
? __list_del_entry_valid.cold+0x1d/0x47
? do_error_trap+0x65/0x80
? __list_del_entry_valid.cold+0x1d/0x47
? exc_invalid_op+0x4e/0x70
? __list_del_entry_valid.cold+0x1d/0x47
? asm_exc_invalid_op+0x16/0x20
? __list_del_entry_valid.cold+0x1d/0x47
lru_gen_del_folio.constprop.0+0x12f/0x1a0
release_pages+0x2dc/0x4e0
__folio_batch_release+0x1b/0x30
truncate_inode_pages_range+0x133/0x3f0
? srso_alias_return_thunk+0x5/0xfbef5
? mdc_lock_match+0xb1/0x180 [mdc]
? srso_alias_return_thunk+0x5/0xfbef5
? ll_have_md_lock+0x158/0x310 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? ldlm_resource_putref+0x30/0x2d0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
ll_lock_cancel_bits+0x444/0x780 [lustre]
ll_md_blocking_ast+0x2dd/0x460 [lustre]
ldlm_cancel_callback+0x7a/0x2b0 [ptlrpc]
ldlm_cli_cancel_local+0xa6/0x490 [ptlrpc]
ldlm_cli_cancel+0xab/0x540 [ptlrpc]
ll_md_blocking_ast+0x320/0x460 [lustre]
ldlm_handle_bl_callback+0x10a/0x3f0 [ptlrpc]
ldlm_bl_thread_blwi.constprop.0+0x7c/0x320 [ptlrpc]
ldlm_bl_thread_main+0x4fb/0x5b0 [ptlrpc]
? __pfx_autoremove_wake_function+0x10/0x10
? __pfx_ldlm_bl_thread_main+0x10/0x10 [ptlrpc]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Modules linked in: nfsv3 rpcrdma rdma_cm iw_cm ib_cm nfsd nfs_acl tls ib_core lustre(OE) mgc(OE) mdc(OE) lov(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) osc(OE) ksocklnd(OE) ptlrpc(OE) obdecho(OE) obdclass(OE) lnet(OE) libcfs(OE) ec(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs rfkill sunrpc intel_rapl_msr intel_rapl_common kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support i2c_i801 pcspkr i2c_smbus lpc_ich virtio_balloon joydev fuse dm_mod drm ext4 mbcache jbd2 ahci libahci crct10dif_pclmul libata crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel net_failover virtio_blk failover serio_raw [last unloaded: libcfs(OE)]
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
LustreError: 137623:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d583c4724000: inode [0x240000bee:0x1:0x0] mdc close failed: rc = -2
LustreError: 139455:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x280000be6:0x133:0x0] : rc = -5
LustreError: 139455:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x280000be6:0x133:0x0] error -5.
LustreError: 139621:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x2c0000be9:0x9e:0x0] : rc = -5
LustreError: 139621:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x2c0000be9:0x9e:0x0] error -5.
Lustre: dir [0x200000bed:0xa7:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 140748:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be8:0x5d:0x0]: rc = -5
LustreError: 140748:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 141682:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d583c4724000: inode [0x280000be7:0x34:0x0] mdc close failed: rc = -2
LustreError: 141683:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d583c4724000: inode [0x240000bee:0x1e:0x0] mdc close failed: rc = -2
4[144470]: segfault at 8 ip 00007f5884807480 sp 00007ffcf61771e8 error 4 in ld-linux-x86-64.so.2[7f58847eb000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
8[144693]: segfault at 8 ip 00007f276185f480 sp 00007ffc1aadd6c8 error 4 in ld-linux-x86-64.so.2[7f2761843000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 145350:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0xa0:0x0]: rc = -5
LustreError: 145350:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 145627:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000bee:0xc3:0x0]: rc = -5
LustreError: 145627:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 145627:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 145627:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 2 previous similar messages
LustreError: 146106:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x2c0000be9:0x9e:0x0] : rc = -5
4[146789]: segfault at 0 ip 0000560a09ba9a00 sp 00007fffb5e78520 error 6 in 4[560a09ba9000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 146789:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d583c4724000: inode [0x200000beb:0xfb:0x0] mdc close failed: rc = -13
LustreError: 146789:0:(file.c:248:ll_close_inode_openhandle()) Skipped 1 previous similar message
Lustre: dir [0x2c0000bea:0xab:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: dir [0x280000be8:0xf1:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
LustreError: 147070:0:(llite_lib.c:1890:ll_update_lsm_md()) lustre: [0x240000bec:0x130:0x0] dir layout mismatch:
LustreError: 147070:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=4 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 147070:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x240000bd1:0x67:0x0]
LustreError: 147070:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=4 migrate_hash=fnv_1a_64:2 pool=
LustreError: 147061:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=4 count=4 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 147061:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=4 migrate_hash=fnv_1a_64:2 pool=
LustreError: 147006:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be8:0x5d:0x0]: rc = -5
LustreError: 147006:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 11 previous similar messages
LustreError: 147006:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 147006:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 11 previous similar messages
Lustre: dir [0x2c0000be9:0x178:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 147680:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d582d8279800: inode [0x240000bef:0x46:0x0] mdc close failed: rc = -2
LustreError: 147680:0:(file.c:248:ll_close_inode_openhandle()) Skipped 2 previous similar messages
LustreError: 149158:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0xe8:0x0]: rc = -5
LustreError: 149158:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 7 previous similar messages
LustreError: 149158:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 149158:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 7 previous similar messages
LustreError: 148853:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x2c0000beb:0x18e:0x0] : rc = -5
LustreError: 148853:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x2c0000beb:0x18e:0x0] error -5.
7[151935]: segfault at 8 ip 00007f789c94b480 sp 00007ffd34e5d638 error 4 in ld-linux-x86-64.so.2[7f789c92f000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 154778:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x2c0000beb:0x18e:0x0] : rc = -5
LustreError: 154778:0:(lov_object.c:1351:lov_layout_change()) Skipped 1 previous similar message
LustreError: 152841:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d582d8279800: inode [0x240000bee:0xf3:0x0] mdc close failed: rc = -2
LustreError: 152841:0:(file.c:248:ll_close_inode_openhandle()) Skipped 5 previous similar messages
LustreError: 151648:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x280000be7:0x167:0x0] error -5.
LustreError: 155985:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000beb:0x16a:0x0]: rc = -5
LustreError: 155985:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 17 previous similar messages
LustreError: 155985:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 155985:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 17 previous similar messages
3[155799]: segfault at 0 ip 000055fb5e961a00 sp 00007ffe60637090 error 6 in 3[55fb5e961000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
6[156051]: segfault at 0 ip 0000555794df3a00 sp 00007fffc69a1560 error 6 in 6[555794df3000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: dir [0x200000beb:0x254:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
LustreError: 156179:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0003-mdc-ff15d582d8279800: dir page locate: [0x2c0000bea:0x7c:0x0] at 0: rc -5
LustreError: 153188:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d583c4724000: cannot apply new layout on [0x280000be7:0x2d3:0x0] : rc = -5
LustreError: 153188:0:(lov_object.c:1351:lov_layout_change()) Skipped 3 previous similar messages
LustreError: 153188:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x280000be7:0x2d3:0x0] error -5.
LustreError: 159875:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0001-mdc-ff15d583c4724000: dir page locate: [0x240000bec:0x242:0x0] at 0: rc -5
LustreError: 159875:0:(mdc_request.c:1511:mdc_read_page()) Skipped 5 previous similar messages
5[163370]: segfault at 8 ip 00007f59c0d56480 sp 00007ffdc60aaa88 error 4 in ld-linux-x86-64.so.2[7f59c0d3a000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 164364:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000beb:0x358:0x0]: rc = -5
LustreError: 164364:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 12 previous similar messages
LustreError: 164364:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 164364:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 12 previous similar messages
LustreError: 166468:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x2c0000bea:0x39c:0x0] : rc = -5
LustreError: 166468:0:(lov_object.c:1351:lov_layout_change()) Skipped 2 previous similar messages
LustreError: 166468:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x2c0000bea:0x39c:0x0] error -5.
LustreError: 166468:0:(vvp_io.c:1915:vvp_io_init()) Skipped 1 previous similar message
Lustre: dir [0x240000bef:0x3df:0x0] stripe 1 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 12 previous similar messages
8[170794]: segfault at 8 ip 00007f4cbf123480 sp 00007ffd280b60a8 error 4 in ld-linux-x86-64.so.2[7f4cbf107000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
3[171039]: segfault at 8 ip 00007f2972b3a480 sp 00007ffe658f4558 error 4 in ld-linux-x86-64.so.2[7f2972b1e000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 174463:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d582d8279800: inode [0x2c0000be8:0x4aa:0x0] mdc close failed: rc = -2
LustreError: 174463:0:(file.c:248:ll_close_inode_openhandle()) Skipped 6 previous similar messages
17[177974]: segfault at 7fdb3abbcd10 ip 000055a0ba35f0c0 sp 00007ffc78daf418 error 7 in 17 (deleted)[55a0ba35c000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 175652:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x240000bef:0x44e:0x0] error -5.
Lustre: dir [0x200000bed:0x832:0x0] stripe 5 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
LustreError: 178363:0:(mdc_request.c:1511:mdc_read_page()) lustre-MDT0002-mdc-ff15d583c4724000: dir page locate: [0x280000bd1:0x86:0x0] at 0: rc -5
LustreError: 178363:0:(mdc_request.c:1511:mdc_read_page()) Skipped 1 previous similar message
1[180905]: segfault at 8 ip 00007fec7f29a480 sp 00007ffca6e20498 error 4 in ld-linux-x86-64.so.2[7fec7f27e000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 136813:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000be8:0x5b6:0x0]: rc = -5
LustreError: 136813:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 59 previous similar messages
LustreError: 136813:0:(llite_lib.c:3738:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 136813:0:(llite_lib.c:3738:ll_prep_inode()) Skipped 59 previous similar messages
15[182556]: segfault at 8 ip 00007fc3b463c480 sp 00007ffda9bbacd8 error 4 in ld-linux-x86-64.so.2[7fc3b4620000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
4[186660]: segfault at 8 ip 00007fc484b2d480 sp 00007ffc7d9f8c08 error 4 in ld-linux-x86-64.so.2[7fc484b11000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
3[190182]: segfault at 8 ip 00007f3a12311480 sp 00007ffc601c2918 error 4 in ld-linux-x86-64.so.2[7f3a122f5000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 189765:0:(lov_object.c:1351:lov_layout_change()) lustre-clilov-ff15d582d8279800: cannot apply new layout on [0x200000bec:0x4ad:0x0] : rc = -5
LustreError: 189765:0:(lov_object.c:1351:lov_layout_change()) Skipped 3 previous similar messages
11[194176]: segfault at 8 ip 00007f1bc00e1480 sp 00007fff2e428fb8 error 4 in ld-linux-x86-64.so.2[7f1bc00c5000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 196796:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff15d583c4724000: inode [0x200000bee:0xac3:0x0] mdc close failed: rc = -2
LustreError: 196796:0:(file.c:248:ll_close_inode_openhandle()) Skipped 11 previous similar messages
19[198362]: segfault at 0 ip 000055d4fd4b0a00 sp 00007ffd96a20f80 error 6 in 19[55d4fd4b0000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
19[203998]: segfault at 8 ip 00007fb79f06c480 sp 00007ffca08ddc78 error 4 in ld-linux-x86-64.so.2[7fb79f050000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
0[204773]: segfault at 8 ip 00007fe67f866480 sp 00007ffe2fa78aa8 error 4 in ld-linux-x86-64.so.2[7fe67f84a000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
13[206999]: segfault at 8 ip 00007f4baf8cf480 sp 00007ffe12362428 error 4 in ld-linux-x86-64.so.2[7f4baf8b3000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
10[209447]: segfault at 1c ip 0000558e8fa3cb90 sp 00007ffc902b3590 error 6 in 10[558e8fa3c000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 211046:0:(vvp_io.c:1915:vvp_io_init()) lustre: refresh file layout [0x280000be6:0xa7b:0x0] error -5.
LustreError: 211046:0:(vvp_io.c:1915:vvp_io_init()) Skipped 2 previous similar messages
LustreError: 210664:0:(llite_lib.c:1890:ll_update_lsm_md()) lustre: [0x200000bec:0xcc7:0x0] dir layout mismatch:
LustreError: 210664:0:(llite_lib.c:1890:ll_update_lsm_md()) Skipped 1 previous similar message
LustreError: 210664:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=4 count=4 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 210664:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x200000bd0:0xa9:0x0]
LustreError: 210664:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 17 previous similar messages
LustreError: 210664:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=4 migrate_hash=fnv_1a_64:2 pool=
Lustre: dir [0x2c0000bea:0x951:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 27 previous similar messages
list_del corruption. next->prev should be ffacaabfc538e348, but was ffacaabfc53d9fc8
------------[ cut here ]------------
Link to test
racer test 1: racer on clients: onyx-157vm57.onyx.whamcloud.com,onyx-157vm58 DURATION=300
kernel BUG at lib/list_debug.c:51!
invalid opcode: 0000 [#1] PREEMPT SMP NOPTI
CPU: 1 PID: 59114 Comm: ldlm_bl_02 Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
RIP: 0010:__list_del_entry_valid.cold+0x31/0x47
Code: 6c 4d b0 e8 fb 69 fe ff 0f 0b 48 c7 c7 d0 6c 4d b0 e8 ed 69 fe ff 0f 0b 48 89 f2 48 89 fe 48 c7 c7 90 6c 4d b0 e8 d9 69 fe ff <0f> 0b 48 89 fe 4c 89 c2 48 c7 c7 58 6c 4d b0 e8 c5 69 fe ff 0f 0b
RSP: 0018:ff69324ac25fba70 EFLAGS: 00010046
RAX: 0000000000000054 RBX: ffb64b7984ff5240 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff4047343bb208c0 RDI: ff4047343bb208c0
RBP: ff4047343ffd3d80 R08: 0000000000000000 R09: ff69324ac25fb930
R10: ff69324ac25fb928 R11: ffffffffb0fe17c8 R12: 0000000000000002
R13: 00000000ffffffff R14: ff404733c6075000 R15: ff69324ac25fbb58
FS: 0000000000000000(0000) GS:ff4047343bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f194c080f20 CR3: 0000000142cde005 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
<TASK>
? srso_alias_return_thunk+0x5/0xfbef5
? show_trace_log_lvl+0x26e/0x2df
? show_trace_log_lvl+0x26e/0x2df
? lru_gen_del_folio.constprop.0+0x12f/0x1a0
? __die_body.cold+0x8/0xd
? die+0x2b/0x50
? do_trap+0xce/0x120
? __list_del_entry_valid.cold+0x31/0x47
? do_error_trap+0x65/0x80
? __list_del_entry_valid.cold+0x31/0x47
? exc_invalid_op+0x4e/0x70
? __list_del_entry_valid.cold+0x31/0x47
? asm_exc_invalid_op+0x16/0x20
? __list_del_entry_valid.cold+0x31/0x47
lru_gen_del_folio.constprop.0+0x12f/0x1a0
release_pages+0x2dc/0x4e0
__folio_batch_release+0x1b/0x30
truncate_inode_pages_range+0x133/0x3f0
? srso_alias_return_thunk+0x5/0xfbef5
? mdc_lock_match+0xb1/0x180 [mdc]
? srso_alias_return_thunk+0x5/0xfbef5
? ll_have_md_lock+0x158/0x310 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? ldlm_resource_putref+0x30/0x2d0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
ll_lock_cancel_bits+0x444/0x780 [lustre]
ll_md_blocking_ast+0x2dd/0x460 [lustre]
ldlm_cancel_callback+0x7a/0x2b0 [ptlrpc]
ldlm_cli_cancel_local+0xa6/0x490 [ptlrpc]
ldlm_cli_cancel+0xab/0x540 [ptlrpc]
ll_md_blocking_ast+0x320/0x460 [lustre]
ldlm_handle_bl_callback+0x10a/0x3f0 [ptlrpc]
ldlm_bl_thread_blwi.constprop.0+0x7c/0x320 [ptlrpc]
ldlm_bl_thread_main+0x4fb/0x5b0 [ptlrpc]
? __pfx_autoremove_wake_function+0x10/0x10
? __pfx_ldlm_bl_thread_main+0x10/0x10 [ptlrpc]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Modules linked in: nfsv3 rpcrdma rdma_cm iw_cm ib_cm nfsd nfs_acl tls ib_core mgc(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ec(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache netfs intel_rapl_msr intel_rapl_common rfkill kvm_amd ccp kvm iTCO_wdt iTCO_vendor_support i2c_i801 i2c_smbus pcspkr virtio_balloon lpc_ich joydev sunrpc fuse drm dm_mod ext4 mbcache jbd2 ahci libahci crct10dif_pclmul crc32_pclmul crc32c_intel libata ghash_clmulni_intel virtio_net net_failover virtio_blk failover serio_raw
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
Lustre: DEBUG MARKER: DURATION=300 MDSCOUNT=4 OSTCOUNT=8 RACER_ENABLE_REMOTE_DIRS=true RACER_ENABLE_STRIPED_DIRS=true RACER_ENABLE_MIGRATION=true RACER_ENABLE_FILE_MIGRATE=true RACER_ENABLE_PFL=true RACER_ENABLE_DOM=true RACER_ENABLE_FLR=true RACER_MA
LustreError: 65511:0:(llite_lib.c:1902:ll_update_lsm_md()) lustre: [0x280000be7:0x62:0x0] dir layout mismatch:
LustreError: 65511:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 65511:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x280000bd1:0x57:0x0]
LustreError: 65511:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 65514:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 65514:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 65508:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 65508:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 65506:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 65506:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 65515:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 65515:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 65702:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=2 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 65702:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=2 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 66315:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x280000be6:0x2:0x0] mdc close failed: rc = -2
LustreError: 67326:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be5:0x76:0x0]: rc = -5
LustreError: 67326:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 66777:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x280000be6:0x12:0x0] mdc close failed: rc = -2
Lustre: dir [0x280000be7:0x91:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
LustreError: 67482:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0001-mdc-ff404733c496f800: dir page locate: [0x240000bec:0x5:0x0] at 0: rc -5
LustreError: 67575:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff404733c496f800: inode [0x200000bed:0x2f:0x0] mdc close failed: rc = -2
LustreError: 68610:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be5:0x76:0x0]: rc = -5
LustreError: 68610:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 2 previous similar messages
LustreError: 68610:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 68610:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 2 previous similar messages
2[69782]: segfault at 8 ip 00007f72846f5480 sp 00007ffd68e45348 error 4 in ld-linux-x86-64.so.2[7f72846d9000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 69241:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x280000be5:0x76:0x0] : rc = -5
LustreError: 69241:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x280000be5:0x76:0x0] error -5.
LustreError: 70288:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x200000bea:0xb2:0x0] : rc = -5
LustreError: 70288:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x200000bea:0xb2:0x0] error -5.
LustreError: 70201:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be5:0x76:0x0]: rc = -5
LustreError: 70201:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 73085:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x240000bee:0x9:0x0] mdc close failed: rc = -2
LustreError: 63972:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x240000bee:0x13:0x0]: rc = -5
LustreError: 63972:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 1 previous similar message
LustreError: 63972:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 63972:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 1 previous similar message
LustreError: 75355:0:(llite_lib.c:1902:ll_update_lsm_md()) lustre: [0x2c0000beb:0x115:0x0] dir layout mismatch:
LustreError: 75355:0:(llite_lib.c:1902:ll_update_lsm_md()) Skipped 5 previous similar messages
LustreError: 75355:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=3 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 75355:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x2c0000bd1:0x48:0x0]
LustreError: 75355:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 41 previous similar messages
LustreError: 75355:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=3 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 75732:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=3 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 75732:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=3 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 75356:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=3 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 75356:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=3 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
LustreError: 75364:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=3 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 75364:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=3 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
7[76936]: segfault at 8 ip 00007f123a89c480 sp 00007ffdc1f2e858 error 4 in ld-linux-x86-64.so.2[7f123a880000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 75764:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x2c0000bec:0x97:0x0] mdc close failed: rc = -2
LustreError: 74505:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x280000be5:0x76:0x0] : rc = -5
LustreError: 74505:0:(lov_object.c:1375:lov_layout_change()) Skipped 1 previous similar message
LustreError: 74505:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x280000be5:0x76:0x0] error -5.
LustreError: 77603:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be5:0x76:0x0]: rc = -5
LustreError: 77603:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 77603:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 77603:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 4 previous similar messages
Lustre: dir [0x2c0000beb:0x151:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 1 previous similar message
LustreError: 76912:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0003-mdc-ff404733c496f800: dir page locate: [0x2c0000bea:0x5d:0x0] at 0: rc -5
Lustre: dir [0x200000bea:0x1d0:0x0] stripe 2 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 3 previous similar messages
15[80276]: segfault at 8 ip 00007fe4df162480 sp 00007ffd9dc79d18 error 4 in ld-linux-x86-64.so.2[7fe4df146000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 79843:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x280000be8:0x12f:0x0] mdc close failed: rc = -2
LustreError: 80400:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff40473429343800: cannot apply new layout on [0x2c0000bec:0x123:0x0] : rc = -5
LustreError: 80400:0:(lov_object.c:1375:lov_layout_change()) Skipped 1 previous similar message
LustreError: 80400:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x2c0000bec:0x123:0x0] error -5.
LustreError: 81624:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x2c0000bec:0x123:0x0]: rc = -5
LustreError: 81624:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 4 previous similar messages
LustreError: 81624:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 81624:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 4 previous similar messages
Lustre: dir [0x280000be7:0x338:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
3[85517]: segfault at 564d9b98f000 ip 0000564d9b98f000 sp 00007ffc5b3cfb18 error 15 in 3[564d9b98f000+1000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <01> 00 02 00 73 6c 65 65 70 00 5b 00 74 65 73 74 20 69 6e 76 6f 63
12[92805]: segfault at 1c ip 000055e715c56b90 sp 00007ffc735ad850 error 6 in 12[55e715c56000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
LustreError: 91935:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x2c0000bec:0x24e:0x0] mdc close failed: rc = -2
LustreError: 91935:0:(file.c:248:ll_close_inode_openhandle()) Skipped 7 previous similar messages
LustreError: 92987:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bed:0x408:0x0]: rc = -5
LustreError: 92987:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 58 previous similar messages
LustreError: 92987:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 92987:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 58 previous similar messages
LustreError: 96294:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x240000bee:0x3c3:0x0] : rc = -5
LustreError: 96294:0:(lov_object.c:1375:lov_layout_change()) Skipped 1 previous similar message
LustreError: 96294:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x240000bee:0x3c3:0x0] error -5.
LustreError: 98850:0:(llite_nfs.c:413:ll_dir_get_parent_fid()) lustre: failure inode [0x200000bed:0x415:0x0] get parent: rc = -2
LustreError: 99932:0:(llite_nfs.c:413:ll_dir_get_parent_fid()) lustre: failure inode [0x2c0000be9:0x280:0x0] get parent: rc = -116
LustreError: 99932:0:(llite_nfs.c:413:ll_dir_get_parent_fid()) Skipped 10 previous similar messages
LustreError: 100931:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x200000beb:0x4f5:0x0] : rc = -5
LustreError: 100931:0:(lov_object.c:1375:lov_layout_change()) Skipped 1 previous similar message
LustreError: 100931:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x200000beb:0x4f5:0x0] error -5.
LustreError: 100931:0:(vvp_io.c:2020:vvp_io_init()) Skipped 1 previous similar message
6[102864]: segfault at 8 ip 00007fafd5420480 sp 00007ffcd7c90648 error 4 in ld-linux-x86-64.so.2[7fafd5404000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 101633:0:(llite_lib.c:1902:ll_update_lsm_md()) lustre: [0x2c0000beb:0x50f:0x0] dir layout mismatch:
LustreError: 101633:0:(llite_lib.c:1902:ll_update_lsm_md()) Skipped 3 previous similar messages
LustreError: 101633:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=1 index=3 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 101633:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x2c0000bd1:0x78:0x0]
LustreError: 101633:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 27 previous similar messages
LustreError: 101633:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=3 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:2000003 pool=
5[103954]: segfault at 8 ip 00007fa80fd78480 sp 00007ffd3a3fb818 error 4 in ld-linux-x86-64.so.2[7fa80fd5c000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
Lustre: dir [0x200000beb:0x6a0:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
LustreError: 110533:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff404733c496f800: inode [0x200000bed:0x634:0x0] mdc close failed: rc = -2
LustreError: 110533:0:(file.c:248:ll_close_inode_openhandle()) Skipped 11 previous similar messages
LustreError: 110815:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x200000bea:0x3a9:0x0]: rc = -5
LustreError: 110815:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 102 previous similar messages
LustreError: 110815:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 110815:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 103 previous similar messages
16[112162]: segfault at 8 ip 00007f9914b70480 sp 00007ffc8aebadb8 error 4 in ld-linux-x86-64.so.2[7f9914b54000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 110784:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff40473429343800: cannot apply new layout on [0x280000be5:0x59a:0x0] : rc = -5
LustreError: 110784:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x280000be5:0x59a:0x0] error -5.
4[114132]: segfault at 55e75d847600 ip 000055e75d847600 sp 00007ffc9766f2d8 error 7 in 17[55e75d847000+4000] likely on CPU 0 (core 0, socket 0)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
17[115180]: segfault at 55893816e600 ip 000055893816e600 sp 00007fff66182de8 error 7 in 17[55893816e000+4000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <00> 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Lustre: dir [0x2c0000bec:0x6d1:0x0] stripe 3 readdir failed: -2, directory is partially accessed!
3[120130]: segfault at 8 ip 00007f707f7e7480 sp 00007ffdb276cc88 error 4 in ld-linux-x86-64.so.2[7f707f7cb000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 121316:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0003-mdc-ff404733c496f800: dir page locate: [0x2c0000bd1:0x92:0x0] at 0: rc -5
LustreError: 121316:0:(mdc_request.c:1510:mdc_read_page()) Skipped 1 previous similar message
LustreError: 57809:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep 17 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 57809:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep 2 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 57809:0:(statahead.c:822:ll_statahead_interpret_work()) Skipped 1 previous similar message
3[127950]: segfault at 2b90 ip 0000000000002b90 sp 00007ffd7a1ea1c0 error 14 likely on CPU 0 (core 0, socket 0)
Code: Unable to access opcode bytes at 0x2b66.
Lustre: dir [0x2c0000bec:0x76a:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 9 previous similar messages
LustreError: 130436:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x200000beb:0xa06:0x0] : rc = -5
LustreError: 130436:0:(lov_object.c:1375:lov_layout_change()) Skipped 8 previous similar messages
LustreError: 129795:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x240000bee:0x8da:0x0] error -5.
LustreError: 129795:0:(vvp_io.c:2020:vvp_io_init()) Skipped 3 previous similar messages
3[132021]: segfault at 8 ip 00007f5fd33cb480 sp 00007fff16fae388 error 4 in ld-linux-x86-64.so.2[7f5fd33af000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 133970:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0003-mdc-ff40473429343800: dir page locate: [0x2c0000bd2:0x36:0x0] at 0: rc -5
LustreError: 133970:0:(mdc_request.c:1510:mdc_read_page()) Skipped 2 previous similar messages
8[139186]: segfault at 8 ip 00007f61fc77c480 sp 00007ffc68f3af68 error 4 in ld-linux-x86-64.so.2[7f61fc760000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 138943:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0003-mdc-ff404733c496f800: dir page locate: [0x2c0000bd2:0x36:0x0] at 0: rc -5
LustreError: 138943:0:(mdc_request.c:1510:mdc_read_page()) Skipped 22 previous similar messages
LustreError: 57809:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep 9 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
3[143415]: segfault at 8 ip 00007f86865a3480 sp 00007ffe7316aa08 error 4 in ld-linux-x86-64.so.2[7f8686587000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
17[148267]: segfault at 8 ip 00007fafb68c8480 sp 00007fff0bde11d8 error 4 in ld-linux-x86-64.so.2[7fafb68ac000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 148759:0:(lcommon_cl.c:186:cl_file_inode_init()) lustre: failed to initialize cl_object [0x280000be8:0xb2f:0x0]: rc = -5
LustreError: 148759:0:(lcommon_cl.c:186:cl_file_inode_init()) Skipped 148 previous similar messages
LustreError: 148759:0:(llite_lib.c:3787:ll_prep_inode()) lustre: new_inode - fatal error: rc = -5
LustreError: 148759:0:(llite_lib.c:3787:ll_prep_inode()) Skipped 148 previous similar messages
LustreError: 149033:0:(file.c:248:ll_close_inode_openhandle()) lustre-clilmv-ff40473429343800: inode [0x280000be5:0xc4d:0x0] mdc close failed: rc = -2
LustreError: 149033:0:(file.c:248:ll_close_inode_openhandle()) Skipped 24 previous similar messages
15[150806]: segfault at 8 ip 00007f5b6ae37480 sp 00007ffe4d3e7908 error 4 in ld-linux-x86-64.so.2[7f5b6ae1b000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
Lustre: dir [0x2c0000beb:0xc52:0x0] stripe 4 readdir failed: -2, directory is partially accessed!
Lustre: Skipped 87 previous similar messages
LustreError: 154617:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0002-mdc-ff404733c496f800: dir page locate: [0x280000be8:0xb60:0x0] at 0: rc -5
LustreError: 154617:0:(mdc_request.c:1510:mdc_read_page()) Skipped 36 previous similar messages
LustreError: 163341:0:(lov_object.c:1375:lov_layout_change()) lustre-clilov-ff404733c496f800: cannot apply new layout on [0x2c0000be9:0xec9:0x0] : rc = -5
LustreError: 163341:0:(lov_object.c:1375:lov_layout_change()) Skipped 25 previous similar messages
LustreError: 163341:0:(vvp_io.c:2020:vvp_io_init()) lustre: refresh file layout [0x2c0000be9:0xec9:0x0] error -5.
LustreError: 163341:0:(vvp_io.c:2020:vvp_io_init()) Skipped 11 previous similar messages
6[168640]: segfault at 8 ip 00007f7d3e799480 sp 00007fffd4c2e448 error 4 in ld-linux-x86-64.so.2[7f7d3e77d000+27000] likely on CPU 0 (core 0, socket 0)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
1[169567]: segfault at 8 ip 00007fefd2cef480 sp 00007ffdfb846f38 error 4 in ld-linux-x86-64.so.2[7fefd2cd3000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
1[169849]: segfault at 8 ip 00007f9de37e2480 sp 00007ffc7ea03f78 error 4 in ld-linux-x86-64.so.2[7f9de37c6000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
14[170050]: segfault at 8 ip 00007f98802ba480 sp 00007ffcb8286a08 error 4 in ld-linux-x86-64.so.2[7f988029e000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
1[171940]: segfault at 556aef092000 ip 0000556aef092000 sp 00007ffc24cb5ee8 error 15 in 1[556aef092000+1000] likely on CPU 1 (core 0, socket 1)
Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <01> 00 02 00 73 6c 65 65 70 00 5b 00 74 65 73 74 20 69 6e 76 6f 63
LustreError: 57865:0:(statahead.c:822:ll_statahead_interpret_work()) lustre: failed to prep 16 [0x0:0x0:0x0] inode@0000000000000000: rc = -5
LustreError: 174468:0:(llite_lib.c:1902:ll_update_lsm_md()) lustre: [0x200000bec:0x1235:0x0] dir layout mismatch:
LustreError: 174468:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 174468:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x200000bd0:0xdb:0x0]
LustreError: 174468:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 5 previous similar messages
LustreError: 174468:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:3 pool=
LustreError: 174475:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 174475:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:3 pool=
LustreError: 174808:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=1 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 174808:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=1 migrate_hash=crush:3 pool=
13[178069]: segfault at 8 ip 00007ff360ca2480 sp 00007ffe3335c548 error 4 in ld-linux-x86-64.so.2[7ff360c86000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 181175:0:(mdc_request.c:1510:mdc_read_page()) lustre-MDT0003-mdc-ff404733c496f800: dir page locate: [0x2c0000bd1:0xe4:0x0] at 0: rc -5
10[185965]: segfault at 8 ip 00007f48c0eff480 sp 00007ffc41bd9fa8 error 4 in ld-linux-x86-64.so.2[7f48c0ee3000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
19[185981]: segfault at 8 ip 00007f970173a480 sp 00007fff94d96768 error 4 in ld-linux-x86-64.so.2[7f970171e000+27000] likely on CPU 1 (core 0, socket 1)
Code: 5b 4c 89 f0 5d 41 5c 41 5d 41 5e c3 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 b8 41 ff ff 6f 29 d0 48 8b 54 c6 40 48 8b 46 68 <48> 8b 40 08 f6 86 1e 03 00 00 20 74 03 48 03 06 48 85 d2 74 2b 48
LustreError: 186491:0:(llite_nfs.c:413:ll_dir_get_parent_fid()) lustre: failure inode [0x200000bec:0x12df:0x0] get parent: rc = -116
LustreError: 186581:0:(llite_lib.c:1902:ll_update_lsm_md()) lustre: [0x200000bed:0x1407:0x0] dir layout mismatch:
LustreError: 186581:0:(llite_lib.c:1902:ll_update_lsm_md()) Skipped 2 previous similar messages
LustreError: 186581:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=3 count=4 index=0 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 186581:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) stripe[0] [0x200000bd0:0xe8:0x0]
LustreError: 186581:0:(lustre_lmv.h:166:lmv_stripe_object_dump()) Skipped 14 previous similar messages
LustreError: 186581:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=5 index=0 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=4 migrate_hash=fnv_1a_64:2 pool=
LustreError: 187911:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=2 count=3 index=1 hash=crush:0x2000003 max_inherit=0 max_inherit_rr=0 version=2 migrate_offset=0 migrate_hash=invalid:0 pool=
LustreError: 187911:0:(lustre_lmv.h:148:lmv_stripe_object_dump()) dump LMV: magic=0xcd20cd0 refs=1 count=4 index=1 hash=crush:0x82000003 max_inherit=0 max_inherit_rr=0 version=1 migrate_offset=3 migrate_hash=fnv_1a_64:2 pool=
list_del corruption. prev->next should be ffb64b7984ff5248, but was ffb64b7984d92588
------------[ cut here ]------------
Link to test
Return to new crashes list