| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanity-lnet test 228: Routes should stay up when health is decremented | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 0 UID: 0 PID: 1622362 Comm: lst Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd out_of_memory.cold+0x61/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0xd7/0x1c0 ? srso_alias_return_thunk+0x5/0xfbef5 pte_alloc_one+0x1e/0x70 __pte_alloc+0x28/0xd0 do_anonymous_page+0x3c5/0x5f0 __handle_mm_fault+0x30f/0x740 handle_mm_fault+0x109/0x350 do_user_addr_fault+0x20c/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7f5d53943624 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: 1621373:0:(lib-socket.c:584:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down LNet: Added LNI fd33:3981:3213:f010:0:5254:69:9d56@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp1 --gateway fd33:3981:3213:f010:0:5254:7f:b61a@tcp LNet: 1621566:0:(router.c:750:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled LNetError: 1621175:0:(lib-lnet.h:1373:lnet_set_route_aliveness()) route to tcp1 through fd33:3981:3213:f010:0:5254:7f:b61a@tcp has gone from up to down Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp1 --gateway fd33:3981:3213:f010:0:5254:27:216e@tcp LNetError: 1621175:0:(lib-lnet.h:1373:lnet_set_route_aliveness()) route to tcp1 through fd33:3981:3213:f010:0:5254:7f:b61a@tcp has gone from down to up Lustre: DEBUG MARKER: /usr/sbin/lctl show_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl route show -v Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:16:c624@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm78.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-156vm78.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1622243:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 500 --all LNet: 2 peer NIs in recovery (showing 2): fd33:3981:3213:f010:0:5254:7f:b61a@tcp, fd33:3981:3213:f010:0:5254:27:216e@tcp Lustre: DEBUG MARKER: Start LST rw Lustre: DEBUG MARKER: Start LST rw lst invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0 CPU: 0 UID: 0 PID: 1622362 Comm: lst Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 dump_header+0x44/0x190 out_of_memory.cold+0x3e/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0xd7/0x1c0 ? srso_alias_return_thunk+0x5/0xfbef5 pte_alloc_one+0x1e/0x70 __pte_alloc+0x28/0xd0 do_anonymous_page+0x3c5/0x5f0 __handle_mm_fault+0x30f/0x740 handle_mm_fault+0x109/0x350 do_user_addr_fault+0x20c/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7f5d53943624 Code: Unable to access opcode bytes at 0x7f5d539435fa. RSP: 002b:00007ffe195c3e50 EFLAGS: 00010206 RAX: 0000000000002fa1 RBX: 0000000000000070 RCX: 000055e3c5600000 RDX: 0000000000000071 RSI: 000055e3c5600060 RDI: 0000000000000004 RBP: 00007f5d53a6dac0 R08: 0000000000000001 R09: 00000000000000ff R10: 0000000000000005 R11: 0000000000000000 R12: 0000000000000000 R13: 000000000000005c R14: 0000000000000070 R15: 00007f5d53a6db20 </TASK> Mem-Info: active_anon:109049 inactive_anon:565879 isolated_anon:0 Node 0 active_anon:437364kB inactive_anon:2262348kB active_file:3472kB inactive_file:3000kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4636kB dirty:0kB writeback:0kB shmem:4552kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:2832kB pagetables:15664kB sec_pagetables:0kB all_unreclaimable? yes Node 0 DMA free:252kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:192kB inactive_anon:14332kB active_file:4kB inactive_file:4kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1625 3543 0 0 Node 0 DMA32 free:30728kB boost:0kB min:30868kB low:38584kB high:46300kB reserved_highatomic:0KB active_anon:617096kB inactive_anon:1030816kB active_file:0kB inactive_file:4kB unevictable:0kB writepending:0kB present:2080624kB managed:1687408kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1918 0 0 Node 0 Normal free:35980kB boost:0kB min:36428kB low:45532kB high:54636kB reserved_highatomic:0KB active_anon:120268kB inactive_anon:917008kB active_file:2536kB inactive_file:3936kB unevictable:0kB writepending:0kB present:2097152kB managed:1970820kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 1*4kB (U) 2*8kB (U) 2*16kB (U) 8*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 308kB Node 0 DMA32: 15*4kB (UM) 23*8kB (UM) 10*16kB (UM) 12*32kB (UM) 15*64kB (UM) 13*128kB (UM) 16*256kB (UM) 15*512kB (U) 16*1024kB (UM) 0*2048kB 0*4096kB = 31572kB Node 0 Normal: 1420*4kB (UM) 290*8kB (UM) 52*16kB (UM) 64*32kB (UM) 34*64kB (UM) 98*128kB (UM) 10*256kB (M) 4*512kB (M) 2*1024kB (M) 2*2048kB (M) 0*4096kB = 36352kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 2601 total pagecache pages 1428 pages in swap cache Free swap = 28kB Total swap = 3807228kB 1048442 pages RAM 0 pages HighMem/MovableOnly 130045 pages reserved 0 pages cma reserved 0 pages hwpoisoned Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 553] 0 553 10301 1307 145 46 1116 98304 192 -250 systemd-journal [ 588] 0 588 3822 71 32 39 0 69632 192 0 systemd-userdbd [ 598] 0 598 8274 567 544 23 0 90112 32 -1000 systemd-udevd [ 632] 32 632 3014 105 64 41 0 65536 128 0 rpcbind [ 635] 0 635 4955 214 182 32 0 57344 0 -1000 auditd [ 694] 81 694 2202 54 32 22 0 61440 128 -900 dbus-broker-lau [ 699] 81 699 1217 132 96 36 0 57344 0 -900 dbus-broker [ 700] 0 700 82310 360 352 8 0 143360 320 0 NetworkManager [ 703] 0 703 20102 8 0 8 0 61440 64 0 irqbalance [ 705] 0 705 4033 173 160 13 0 77824 96 0 systemd-logind [ 713] 996 713 21056 74 42 32 0 65536 96 0 chronyd [ 727] 0 727 30519 477 448 29 0 98304 64 0 gssproxy [ 728] 0 728 2017 86 64 22 0 49152 160 -1000 sshd [ 795] 0 795 44850 789 160 45 584 106496 160 0 rsyslogd [ 798] 29 798 2506 112 80 32 0 53248 128 0 rpc.statd [ 801] 998 801 51813 121 108 13 0 69632 160 0 munged [ 854] 0 854 84147 378 352 26 0 126976 128 0 automount [ 867] 0 867 1245 26 0 26 0 49152 32 0 agetty [ 868] 0 868 1887 61 32 29 0 61440 0 0 agetty [ 895] 0 895 2070 185 160 25 0 61440 64 0 crond [ 1098] 0 1098 3608 415 384 31 0 69632 0 0 sshd-session [ 1103] 0 1103 4993 488 448 40 0 73728 0 100 systemd [ 1105] 0 1105 3263 86 64 22 0 61440 224 100 (sd-pam) [ 1114] 0 1114 3720 367 309 58 0 69632 192 0 sshd-session [ 1480] 0 1480 1737 27 0 27 0 49152 64 0 run_test_wrappe [ 1520] 0 1520 1770 32 0 32 0 49152 64 0 run_test.sh [ 1852] 0 1852 3720 62 0 62 0 69632 1952 0 bash [1209762] 0 1209762 3608 434 384 50 0 65536 0 0 sshd-session [1209783] 0 1209783 3845 57 8 49 0 65536 704 0 ssh [1209790] 0 1209790 3322 23 18 5 0 65536 576 0 ssh [1209791] 0 1209791 3350 37 0 37 0 69632 704 0 ssh [1209793] 0 1209793 3255 21 0 21 0 69632 576 0 ssh [1209795] 0 1209795 3446 66 30 36 0 65536 768 0 ssh [1209796] 0 1209796 3690 499 470 29 0 65536 0 0 sshd-session [1513974] 0 1513974 3720 62 22 40 0 65536 1952 0 bash [1513975] 0 1513975 708 30 0 30 0 49152 0 0 tee [1514222] 0 1514222 3396 52 39 13 0 69632 2368 0 bash [1611689] 0 1611689 3933 49 32 17 0 77824 192 0 systemd-userwor [1619810] 0 1619810 3396 42 9 33 0 69632 2336 0 bash [1619811] 0 1619811 708 23 0 23 0 53248 0 0 tee [1621293] 0 1621293 3933 184 160 24 0 73728 64 0 systemd-userwor [1622249] 0 1622249 1172 54 32 22 0 53248 96 0 lst.sh [1622362] 0 1622362 1605163 666806 666784 22 0 12906496 937280 0 lst [1622363] 0 1622363 3933 248 224 24 0 73728 0 0 systemd-userwor | Link to test (2026-10-23 04:00) |