Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 0 UID: 0 PID: 718 Comm: gssproxy Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1
Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
panic+0x113/0x2dd
out_of_memory.cold+0x61/0x78
__alloc_pages_slowpath.constprop.0+0x594/0xb70
__alloc_pages_noprof+0x31b/0x330
alloc_pages_mpol_noprof+0xd7/0x1c0
folio_alloc_noprof+0x5b/0xa0
__filemap_get_folio+0x19f/0x2e0
filemap_fault+0x1fb/0x9b0
__do_fault+0x30/0x180
do_read_fault+0x11e/0x1f0
do_fault+0x200/0x330
__handle_mm_fault+0x30f/0x740
? srso_alias_return_thunk+0x5/0xfbef5
handle_mm_fault+0x109/0x350
do_user_addr_fault+0x20c/0x640
exc_page_fault+0x73/0x160
asm_exc_page_fault+0x26/0x30
RIP: 0033:0x7f7941fd6295

Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp --if eth0
LNet: 1622011:0:(lib-socket.c:584:lnet_inet_enumerate()) lnet: Ignoring interface test1pg: it's down
LNet: Added LNI fd33:3981:3213:f020:0:5254:1c:602e@tcp [8/256/0/180]
Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp1 --gateway fd33:3981:3213:f020:0:5254:f1:4fd8@tcp
LNet: 1622204:0:(router.c:750:lnet_add_route()) Use hops = 1 for a single-hop route when avoid_asym_router_failure feature is enabled
LNetError: 1621812:0:(lib-lnet.h:1373:lnet_set_route_aliveness()) route to tcp1 through fd33:3981:3213:f020:0:5254:f1:4fd8@tcp has gone from up to down
Lustre: DEBUG MARKER: /usr/sbin/lnetctl route add --net tcp1 --gateway fd33:3981:3213:f020:0:5254:fe:b8db@tcp
LNetError: 1621812:0:(lib-lnet.h:1373:lnet_set_route_aliveness()) route to tcp1 through fd33:3981:3213:f020:0:5254:f1:4fd8@tcp has gone from down to up
Lustre: DEBUG MARKER: /usr/sbin/lctl show_route
Lustre: DEBUG MARKER: /usr/sbin/lnetctl route show -v
Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:83:f436@tcp1
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm20.trevis.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest
Lustre: DEBUG MARKER: trevis-156vm20.trevis.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
LNet: 1622881:0:(rpc.c:645:srpc_service_add_buffers()) waiting for adding buffer
Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 500 --all
Lustre: DEBUG MARKER: Start LST rw
Lustre: DEBUG MARKER: Start LST rw
LNet: 1 peer NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:fe:b8db@tcp
gssproxy invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0
CPU: 0 UID: 0 PID: 718 Comm: gssproxy Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1
Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
dump_header+0x44/0x190
out_of_memory.cold+0x3e/0x78
__alloc_pages_slowpath.constprop.0+0x594/0xb70
__alloc_pages_noprof+0x31b/0x330
alloc_pages_mpol_noprof+0xd7/0x1c0
folio_alloc_noprof+0x5b/0xa0
__filemap_get_folio+0x19f/0x2e0
filemap_fault+0x1fb/0x9b0
__do_fault+0x30/0x180
do_read_fault+0x11e/0x1f0
do_fault+0x200/0x330
__handle_mm_fault+0x30f/0x740
? srso_alias_return_thunk+0x5/0xfbef5
handle_mm_fault+0x109/0x350
do_user_addr_fault+0x20c/0x640
exc_page_fault+0x73/0x160
asm_exc_page_fault+0x26/0x30
RIP: 0033:0x7f7941fd6295
Code: Unable to access opcode bytes at 0x7f7941fd626b.
RSP: 002b:00007ffc8d79a120 EFLAGS: 00010206
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 0000000001863af4
RDX: 00000000000056b4 RSI: 00007ffc8d79a140 RDI: 00007f7942bd8090
RBP: 0000000000000000 R08: 000000000010bc90 R09: 0000000000000000
R10: 0000000000000000 R11: 00007f7942bd8080 R12: 41dab6b0d677639c
R13: 0000000000000009 R14: 0000000000000000 R15: 00007f7941fde060
</TASK>
Mem-Info:
active_anon:282947 inactive_anon:390987 isolated_anon:0
Node 0 active_anon:1131788kB inactive_anon:1563948kB active_file:4604kB inactive_file:3720kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4848kB dirty:68kB writeback:0kB shmem:4716kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:2896kB pagetables:15644kB sec_pagetables:0kB all_unreclaimable? no
Node 0 DMA free:276kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:1472kB inactive_anon:13024kB active_file:0kB inactive_file:4kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1571 3543 0 0
Node 0 DMA32 free:30748kB boost:0kB min:29840kB low:37300kB high:44760kB reserved_highatomic:0KB active_anon:681708kB inactive_anon:916976kB active_file:352kB inactive_file:544kB unevictable:0kB writepending:0kB present:2080624kB managed:1638664kB mlocked:0kB bounce:0kB free_pcp:64kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1972 0 0
Node 0 Normal free:37196kB boost:0kB min:37456kB low:46820kB high:56184kB reserved_highatomic:0KB active_anon:448272kB inactive_anon:633948kB active_file:4084kB inactive_file:3732kB unevictable:0kB writepending:68kB present:2097152kB managed:2019556kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 0*4kB 0*8kB 5*16kB (U) 8*32kB (U) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 336kB
Node 0 DMA32: 1*4kB (M) 12*8kB (U) 5*16kB (UM) 11*32kB (UM) 12*64kB (UM) 13*128kB (UM) 14*256kB (UM) 18*512kB (UM) 15*1024kB (UM) 0*2048kB 0*4096kB = 31124kB
Node 0 Normal: 1822*4kB (UM) 451*8kB (UM) 75*16kB (UM) 62*32kB (UM) 12*64kB (UM) 25*128kB (UM) 56*256kB (UM) 11*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 38016kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
2941 total pagecache pages
1591 pages in swap cache
Free swap = 20kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
130047 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 545] 0 545 10300 1151 13 31 1107 86016 288 -250 systemd-journal
[ 583] 0 583 3822 232 192 40 0 69632 0 0 systemd-userdbd
[ 592] 0 592 8250 543 512 31 0 81920 0 -1000 systemd-udevd
[ 622] 32 622 3014 165 128 37 0 65536 64 0 rpcbind
[ 626] 0 626 4955 179 148 31 0 49152 64 -1000 auditd
[ 683] 81 683 2202 67 32 35 0 57344 128 -900 dbus-broker-lau
[ 686] 81 686 1219 120 64 56 0 49152 32 -900 dbus-broker
[ 687] 0 687 82313 668 640 28 0 139264 0 0 NetworkManager
[ 690] 0 690 20102 43 32 11 0 57344 32 0 irqbalance
[ 694] 0 694 4032 175 160 15 0 77824 128 0 systemd-logind
[ 710] 996 710 21056 63 39 24 0 69632 96 0 chronyd
[ 718] 0 718 30519 470 448 22 0 110592 32 0 gssproxy
[ 722] 0 722 2017 278 224 54 0 65536 0 -1000 sshd
[ 780] 0 780 44850 861 288 8 565 106496 0 0 rsyslogd
[ 782] 29 782 2506 73 48 25 0 53248 128 0 rpc.statd
[ 786] 998 786 51813 131 108 23 0 86016 224 0 munged
[ 829] 0 829 84148 462 416 46 0 118784 64 0 automount
[ 841] 0 841 1245 30 0 30 0 53248 32 0 agetty
[ 842] 0 842 1887 88 32 56 0 61440 0 0 agetty
[ 867] 0 867 2070 109 64 45 0 53248 128 0 crond
[ 1072] 0 1072 3608 403 384 19 0 73728 0 0 sshd-session
[ 1077] 0 1077 4992 492 448 44 0 73728 0 100 systemd
[ 1079] 0 1079 3263 156 119 37 0 57344 160 100 (sd-pam)
[ 1088] 0 1088 3726 305 276 29 0 77824 192 0 sshd-session
[ 1451] 0 1451 1737 21 0 21 0 53248 64 0 run_test_wrappe
[ 1491] 0 1491 1770 0 0 0 0 61440 64 0 run_test.sh
[ 1820] 0 1820 3720 49 22 27 0 73728 1920 0 bash
[1209690] 0 1209690 3608 391 352 39 0 65536 0 0 sshd-session
[1209711] 0 1209711 3306 70 23 47 0 69632 608 0 ssh
[1209714] 0 1209714 3534 100 51 49 0 65536 832 0 ssh
[1209718] 0 1209718 3324 94 55 39 0 69632 640 0 ssh
[1209722] 0 1209722 3479 65 44 21 0 69632 768 0 ssh
[1209723] 0 1209723 4055 61 36 25 0 73728 864 0 ssh
[1209724] 0 1209724 3690 479 479 0 0 65536 0 0 sshd-session
[1514537] 0 1514537 3720 45 22 23 0 69632 1952 0 bash
[1514538] 0 1514538 708 23 0 23 0 45056 0 0 tee
[1514785] 0 1514785 3396 55 40 15 0 65536 2368 0 bash
[1613585] 0 1613585 3933 276 224 52 0 69632 0 0 systemd-userwor
[1616757] 0 1616757 3933 273 224 49 0 73728 0 0 systemd-userwor
[1620447] 0 1620447 3396 77 42 35 0 65536 2336 0 bash
[1620448] 0 1620448 708 30 0 30 0 45056 0 0 tee
[1621969] 0 1621969 3933 269 224 45 0 73728 0 0 systemd-userwor
[1622887] 0 1622887 1172 80 32 48 0 49152 96 0 lst.sh
[1623000] 0 1623000 1603546 665079 664992 87 0 12894208 937408 0 lst