| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| racer test 1: racer on clients: centos-10.localnet DURATION=2700 | general protection fault, probably for non-canonical address 0x481af07bfc50207a: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 1 PID: 33695 Comm: rm Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #1 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 RIP: 0010:memset_orig+0x94/0xb0 Code: 66 0f 1f 44 00 00 ff c9 48 89 07 48 8d 7f 08 75 f5 83 e2 07 74 0a ff ca 88 07 48 8d 7f 01 75 f6 4c 89 d0 c3 48 83 fa 07 76 e7 <48> 89 07 49 c7 c0 08 00 00 00 4d 29 c8 4c 01 c7 4c 29 c2 e9 72 ff RSP: 0018:ffffbde6485efb88 EFLAGS: 00010216 RAX: 0000000000000000 RBX: d25ad1d13439c48d RCX: 0000000000000000 RDX: 0000000000000160 RSI: 0000000000000000 RDI: 481af07bfc50207a RBP: ffff99b34684cc40 R08: ffffffff8ce2cca0 R09: 0000000000000002 R10: 481af07bfc50207a R11: 0000000000000001 R12: 0000000000608040 R13: ffff99b34684cc40 R14: ffffffffc1105c51 R15: 000000000000277a FS: 00007f50ee7ef640(0000) GS:ffff99b572240000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007f4e9f286008 CR3: 0000000201828000 CR4: 00000000000006e0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? die_addr+0x50/0x90 ? do_general_protection+0x208/0x420 ? general_protection+0x1e/0x30 ? osc_session_init+0x31/0x150 [osc] ? memset_orig+0x94/0xb0 ? osc_session_init+0x31/0x150 [osc] kmem_cache_alloc+0x307/0x430 ? lov_sub_get+0x1e1/0xbb0 [lov] osc_session_init+0x31/0x150 [osc] keys_fill+0xac/0x1c0 [obdclass] lu_context_init+0xcc/0x230 [obdclass] cl_env_new+0x223/0x2e0 [obdclass] ? lov_sub_get+0x1e1/0xbb0 [lov] cl_env_get+0xe3/0x3a0 [obdclass] lov_sub_get+0x1e1/0xbb0 [lov] lov_io_iter_init+0x418/0xd00 [lov] cl_io_iter_init+0x5e/0x190 [obdclass] cl_io_loop+0x95/0x2f0 [obdclass] cl_sync_file_range+0x40c/0x4b0 [lustre] ll_delete_inode+0x193/0x1c0 [lustre] evict+0x119/0x330 iput+0x22f/0x3b0 do_unlinkat+0x343/0x460 __x64_sys_unlinkat+0x4c/0xa0 do_syscall_64+0xc1/0x3f0 entry_SYSCALL_64_after_hwframe+0x49/0xae RIP: 0033:0x7f50ee22313b | Lustre: 9011:0:(mdt_recovery.c:102:mdt_req_from_lrd()) @@@ restoring transno req@ffff99b42145b100 x1860942675673088/t4294970539(0) o101->031357c0-6bb1-43b2-9bfb-28deaa4bd0c7@0@lo:112/0 lens 376/840 e 0 to 0 dl 1774733312 ref 1 fl Interpret:H/602/0 rc 0/0 job:'dd.0' uid:0 gid:0 projid:0 LustreError: 13979:0:(file.c:251:ll_close_inode_openhandle()) lustre-clilmv-ffff99b3e310e000: inode [0x200000401:0x1f7:0x0] mdc close failed: rc = -116 Lustre: 6054:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0000: opcode 7: before 516 < left 618, rollback = 7 Lustre: 6054:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 6054:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 Lustre: 6054:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0 Lustre: 6054:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 Lustre: 6054:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 Lustre: 5793:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 514 < left 520, rollback = 2 Lustre: 5793:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 5/20/0, destroy: 1/4/0 Lustre: 5793:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 5793:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 17/17/0, xattr_set: 26/520/0 Lustre: 5793:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 5793:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 24/144/0, punch: 0/0/0, quota 7/249/0 Lustre: 5793:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 5793:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 7/117/1, delete: 3/6/1 Lustre: 5793:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 5793:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 1/1/0 Lustre: 5793:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message 3[22426]: segfault at 68 ip 000056518b38df94 sp 00007ffcc25554b8 error 6 in 3[56518b389000+7000] Code: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 61 32 35 36 73 75 6d <00> 73 68 61 33 38 34 73 75 6d 00 73 68 61 35 31 32 73 75 6d 00 0a Lustre: 6054:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7 Lustre: 6054:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message Lustre: 6054:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 6054:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 Lustre: 6054:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 1/3/0 Lustre: 6054:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 Lustre: 6054:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 Lustre: 9221:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7 Lustre: 9221:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 3 previous similar messages Lustre: 9221:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 9221:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 9221:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 Lustre: 9221:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 9221:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0 Lustre: 9221:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 9221:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 Lustre: 9221:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 9221:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 Lustre: 9221:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 6056:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0003: opcode 7: before 516 < left 618, rollback = 7 Lustre: 6056:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 6056:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 Lustre: 6056:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0 Lustre: 6056:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 Lustre: 6056:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 Lustre: 6056:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message LustreError: 5769:0:(ldlm_lockd.c:255:expired_lock_main()) ### lock callback timer expired after 101s: evicting client at 0@lo ns: filter-lustre-OST0001_UUID lock: 000000000070b051/0x37cd1c2445428bee lrc: 3/0,0 mode: PW/PW res: [0x280000400:0x3:0x0].0x0 rrc: 5 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x60000000020020 nid: 0@lo remote: 0x37cd1c2445428b5b expref: 13 pid: 6051 timeout: 192 lvb_type: 0 lru_score: 0 lru_type: 0 LustreError: lustre-OST0001-osc-ffff99b3e310e000: operation ost_setattr to node 0@lo failed: rc = -107 Lustre: lustre-OST0001-osc-ffff99b3e310e000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: lustre-OST0001-osc-ffff99b3e310e000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: 3207:0:(llite_lib.c:4356:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x61f:0x0]/ may get corrupted (rc -108) Lustre: 3195:0:(llite_lib.c:4356:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x846:0x0]/ may get corrupted (rc -108) Lustre: 3209:0:(llite_lib.c:4356:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000402:0x7:0x0]// may get corrupted (rc -108) Lustre: 3196:0:(llite_lib.c:4356:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x855:0x0]/ may get corrupted (rc -108) Lustre: 3210:0:(llite_lib.c:4356:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.12@tcp:/lustre/fid: [0x200000401:0x6c8:0x0]/ may get corrupted (rc -108) LustreError: 30413:0:(ldlm_resource.c:1170:ldlm_resource_complain()) lustre-OST0001-osc-ffff99b3e310e000: namespace resource [0x280000400:0x3:0x0].0x0 (ffff99b354850400) refcount nonzero (2) after lock cleanup; forcing cleanup. Lustre: lustre-OST0001-osc-ffff99b3e310e000: Connection restored to 0@lo (at 0@lo) Lustre: 6056:0:(osd_internal.h:1446:osd_trans_exec_op()) lustre-OST0001: opcode 7: before 516 < left 618, rollback = 7 Lustre: 6056:0:(osd_internal.h:1446:osd_trans_exec_op()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 6056:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/15/0 Lustre: 6056:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/618/0, punch: 0/0/0, quota 4/150/0 Lustre: 6056:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 Lustre: 6056:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message Lustre: 6056:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 Lustre: 6056:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message | Link to test |