| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanity-lnet test 228: Routes should stay up when health is decremented | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 1 UID: 0 PID: 1618629 Comm: lst Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd out_of_memory.cold+0x61/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0xd7/0x1c0 ? srso_alias_return_thunk+0x5/0xfbef5 pte_alloc_one+0x1e/0x70 __pte_alloc+0x28/0xd0 do_anonymous_page+0x3c5/0x5f0 __handle_mm_fault+0x30f/0x740 handle_mm_fault+0x109/0x350 do_user_addr_fault+0x20c/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7f1032421624 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: 1617640:0:(lib-socket.c:584:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down LNet: Added LNI fd33:3981:3213:f010:0:5254:6d:feed@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp1 --gateway fd33:3981:3213:f010:0:5254:87:b2f3@tcp LNet: 1617833:0:(router.c:750:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled LNetError: 1617443:0:(lib-lnet.h:1373:lnet_set_route_aliveness()) route to tcp1 through fd33:3981:3213:f010:0:5254:87:b2f3@tcp has gone from up to down Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp1 --gateway fd33:3981:3213:f010:0:5254:ee:c407@tcp LNetError: 1617443:0:(lib-lnet.h:1373:lnet_set_route_aliveness()) route to tcp1 through fd33:3981:3213:f010:0:5254:87:b2f3@tcp has gone from down to up Lustre: DEBUG MARKER: /usr/sbin/lctl show_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl route show -v Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:5d:8ff6@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-158vm82.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-158vm82.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer LNet: 1618510:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 500 --all LNet: 2 peer NIs in recovery (showing 2): fd33:3981:3213:f010:0:5254:ee:c407@tcp, fd33:3981:3213:f010:0:5254:87:b2f3@tcp Lustre: DEBUG MARKER: Start LST rw Lustre: DEBUG MARKER: Start LST rw lst invoked oom-killer: gfp_mask=0x440dc0(GFP_KERNEL_ACCOUNT|__GFP_COMP|__GFP_ZERO), order=0, oom_score_adj=0 CPU: 1 UID: 0 PID: 1618629 Comm: lst Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 dump_header+0x44/0x190 out_of_memory.cold+0x3e/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0xd7/0x1c0 ? srso_alias_return_thunk+0x5/0xfbef5 pte_alloc_one+0x1e/0x70 __pte_alloc+0x28/0xd0 do_anonymous_page+0x3c5/0x5f0 __handle_mm_fault+0x30f/0x740 handle_mm_fault+0x109/0x350 do_user_addr_fault+0x20c/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7f1032421624 Code: Unable to access opcode bytes at 0x7f10324215fa. RSP: 002b:00007fff234c0e10 EFLAGS: 00010206 RAX: 0000000000016fa1 RBX: 0000000000000070 RCX: 00005569c5000000 RDX: 0000000000000071 RSI: 00005569c5000060 RDI: 0000000000000004 RBP: 00007f103254bac0 R08: 0000000000000001 R09: 00000000000000ff R10: 0000000000000005 R11: 0000000000000000 R12: 0000000000000000 R13: 000000000000005c R14: 0000000000000070 R15: 00007f103254bb20 </TASK> Mem-Info: active_anon:324534 inactive_anon:350184 isolated_anon:0 Node 0 active_anon:1297208kB inactive_anon:1401664kB active_file:2880kB inactive_file:4516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:2924kB dirty:0kB writeback:0kB shmem:2832kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:2896kB pagetables:15648kB sec_pagetables:0kB all_unreclaimable? yes Node 0 DMA free:336kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:3172kB inactive_anon:11324kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1571 3543 0 0 Node 0 DMA32 free:30912kB boost:0kB min:29840kB low:37300kB high:44760kB reserved_highatomic:0KB active_anon:892856kB inactive_anon:707808kB active_file:0kB inactive_file:336kB unevictable:0kB writepending:0kB present:2080624kB managed:1638664kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1972 0 0 Node 0 Normal free:37140kB boost:0kB min:37456kB low:46820kB high:56184kB reserved_highatomic:0KB active_anon:401180kB inactive_anon:682532kB active_file:5232kB inactive_file:2164kB unevictable:0kB writepending:0kB present:2097152kB managed:2019556kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 2*4kB (U) 1*8kB (U) 4*16kB (U) 8*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 336kB Node 0 DMA32: 35*4kB (M) 61*8kB (UM) 48*16kB (UM) 23*32kB (UM) 14*64kB (UM) 11*128kB (UM) 14*256kB (U) 15*512kB (UM) 15*1024kB (UM) 0*2048kB 0*4096kB = 31060kB Node 0 Normal: 828*4kB (UM) 273*8kB (UM) 73*16kB (UM) 44*32kB (UM) 16*64kB (M) 11*128kB (UM) 11*256kB (UM) 29*512kB (M) 9*1024kB (UM) 0*2048kB 0*4096kB = 37384kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 1997 total pagecache pages 1242 pages in swap cache Free swap = 172kB Total swap = 3807228kB 1048442 pages RAM 0 pages HighMem/MovableOnly 130047 pages reserved 0 pages cma reserved 0 pages hwpoisoned Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 550] 0 550 10303 725 9 15 701 81920 320 -250 systemd-journal [ 585] 0 585 3822 5 0 5 0 69632 192 0 systemd-userdbd [ 595] 0 595 8255 349 288 61 0 86016 256 -1000 systemd-udevd [ 612] 32 612 3014 41 0 41 0 69632 192 0 rpcbind [ 617] 0 617 4955 54 16 38 0 53248 192 -1000 auditd [ 688] 81 688 2202 54 32 22 0 57344 128 -900 dbus-broker-lau [ 691] 81 691 1220 151 96 55 0 45056 0 -900 dbus-broker [ 692] 0 692 82310 435 416 19 0 139264 256 0 NetworkManager [ 695] 0 695 20102 69 32 37 0 61440 64 0 irqbalance [ 697] 0 697 4036 70 32 38 0 73728 224 0 systemd-logind [ 699] 996 699 21056 46 8 38 0 65536 160 0 chronyd [ 719] 0 719 30519 247 192 55 0 94208 320 0 gssproxy [ 724] 0 724 2017 96 64 32 0 61440 160 -1000 sshd [ 788] 0 788 44850 818 192 26 600 106496 96 0 rsyslogd [ 789] 29 789 2506 32 16 16 0 53248 160 0 rpc.statd [ 793] 998 793 51813 131 108 23 0 69632 160 0 munged [ 850] 0 850 84150 290 256 34 0 118784 224 0 automount [ 863] 0 863 1245 39 0 39 0 53248 32 0 agetty [ 864] 0 864 1887 9 0 9 0 53248 64 0 agetty [ 888] 0 888 2070 19 0 19 0 57344 192 0 crond [ 1093] 0 1093 3608 401 384 17 0 69632 0 0 sshd-session [ 1098] 0 1098 4993 487 448 39 0 81920 0 100 systemd [ 1100] 0 1100 3263 21 0 21 0 57344 288 100 (sd-pam) [ 1109] 0 1109 3687 318 308 10 0 69632 192 0 sshd-session [ 1475] 0 1475 1737 68 32 36 0 53248 64 0 run_test_wrappe [ 1515] 0 1515 1770 43 0 43 0 57344 64 0 run_test.sh [ 1846] 0 1846 3720 71 22 49 0 69632 1952 0 bash [1205557] 0 1205557 3608 415 384 31 0 77824 0 0 sshd-session [1205578] 0 1205578 3128 32 0 32 0 69632 480 0 ssh [1205582] 0 1205582 3302 59 15 44 0 65536 640 0 ssh [1205584] 0 1205584 4247 49 36 13 0 73728 608 0 ssh [1205589] 0 1205589 3426 49 6 43 0 69632 704 0 ssh [1205590] 0 1205590 3145 56 23 33 0 65536 480 0 ssh [1205591] 0 1205591 3690 307 276 31 0 77824 192 0 sshd-session [1510232] 0 1510232 3720 39 15 24 0 65536 1952 0 bash [1510233] 0 1510233 708 35 0 35 0 49152 32 0 tee [1510480] 0 1510480 3392 29 12 17 0 69632 2368 0 bash [1608907] 0 1608907 3933 183 128 55 0 77824 96 0 systemd-userwor [1611072] 0 1611072 3933 274 224 50 0 65536 0 0 systemd-userwor [1611073] 0 1611073 3933 241 224 17 0 77824 0 0 systemd-userwor [1616076] 0 1616076 3392 51 10 41 0 69632 2368 0 bash [1616077] 0 1616077 708 13 0 13 0 49152 0 0 tee [1618516] 0 1618516 1172 33 0 33 0 49152 96 0 lst.sh [1618629] 0 1618629 1605526 669203 669184 19 0 12910592 935200 0 lst | Link to test (2026-10-23 05:25) |