Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.7.1.el9_7.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
panic+0x107/0x2bb
out_of_memory.cold+0x5e/0x7e
__alloc_pages_slowpath.constprop.0+0x793/0xb10
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1e0
? rmqueue_bulk+0x348/0x9c0
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? virtqueue_add_split+0x365/0x720
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x41a/0xa50
? __pte_offset_map+0x17/0x140
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x620
exc_page_fault+0x61/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0010:__put_user_nocheck_4+0x3/0x10

Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 7243604kB available, using 7482080kB file size
Lustre: DEBUG MARKER: min OST has 7243604kB available, using 7482080kB file size
Autotest: Test running for 210 minutes (lustre-master-next_full-part-3_1018.174) | 2026-10-22 06:12:31
LNet: 124539:0:(debug.c:354:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
Autotest: Test running for 215 minutes (lustre-master-next_full-part-3_1018.174) | 2026-10-22 06:17:31
LNet: 125493:0:(debug.c:354:libcfs_debug_str2mask()) You are trying to use a numerical value for the mask - this will be deprecated in a future release.
Lustre: 3116:0:(client.c:2289:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792649954/real 1792649954] req@000000002abc4f8f x1879718872136832/t0(0) o4->lustre-OST0000-osc-ff3cda1503b7c800@10.240.23.219@tcp:6/4 lens 488/448 e 0 to 1 dl 1792649966 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'iozone.500'
Lustre: 3116:0:(client.c:2289:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST0000-osc-ff3cda1503b7c800: Connection to lustre-OST0000 (at 10.240.23.219@tcp) was lost; in progress operations using this service will wait for recovery to complete
LustreError: 3108:0:(events.c:211:client_bulk_callback()) event type 1, status -5, desc 00000000fffac2a9
Lustre: lustre-OST0000-osc-ff3cda1503b7c800: Connection restored to (at 10.240.23.219@tcp)
Lustre: Skipped 3 previous similar messages
Lustre: 3114:0:(client.c:2289:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792649993/real 1792649993] req@0000000061f61db1 x1879718872137216/t0(0) o4->lustre-OST0004-osc-ff3cda1503b7c800@10.240.23.219@tcp:6/4 lens 488/448 e 0 to 1 dl 1792650005 ref 2 fl Rpc:XQr/2/ffffffff rc 0/-1 job:'iozone.500'
Lustre: 3114:0:(client.c:2289:ptlrpc_expire_one_request()) Skipped 7 previous similar messages
Lustre: lustre-OST0004-osc-ff3cda1503b7c800: Connection to lustre-OST0004 (at 10.240.23.219@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 5 previous similar messages
Lustre: lustre-OST0003-osc-ff3cda1503b7c800: Connection restored to (at 10.240.23.219@tcp)
Lustre: Skipped 5 previous similar messages
Lustre: 3114:0:(client.c:2289:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1792650076/real 0] req@00000000948f5dee x1879718874679168/t0(0) o400->lustre-OST0001-osc-ff3cda1503b7c800@10.240.23.219@tcp:28/4 lens 224/224 e 0 to 1 dl 1792650088 ref 2 fl Rpc:XNr/0/ffffffff rc 0/-1 job:'kworker/u9:1.0'
Lustre: 3114:0:(client.c:2289:ptlrpc_expire_one_request()) Skipped 2 previous similar messages
Lustre: lustre-OST0001-osc-ff3cda1503b7c800: Connection to lustre-OST0001 (at 10.240.23.219@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 2 previous similar messages
LustreError: 166-1: MGC10.240.23.220@tcp: Connection to MGS (at 10.240.23.220@tcp) was lost; in progress operations using this service will fail
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
obd_memory max: 3010640727, obd_memory current: 45220807
systemd invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.7.1.el9_7.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
dump_header+0x49/0x212
out_of_memory.cold+0x39/0x7e
__alloc_pages_slowpath.constprop.0+0x793/0xb10
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1e0
? rmqueue_bulk+0x348/0x9c0
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? virtqueue_add_split+0x365/0x720
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x41a/0xa50
? __pte_offset_map+0x17/0x140
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x620
exc_page_fault+0x61/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0010:__put_user_nocheck_4+0x3/0x10
Code: d9 0f 01 cb 89 01 31 c9 0f 01 ca e9 82 12 81 00 0f 1f 80 00 00 00 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 0f 01 cb <89> 01 31 c9 0f 01 ca e9 5c 12 81 00 90 90 90 90 90 90 90 90 90 90
RSP: 0018:ff507ec800013d08 EFLAGS: 00050202
RAX: 0000000000000001 RBX: 0000000000000000 RCX: 000055b1acd93370
RDX: 0000000000000001 RSI: 0000000000000001 RDI: ff3cda150670fa00
RBP: ff3cda1506dc6618 R08: ff507ec800013d28 R09: ff3cda151d4e4768
R10: 00000000000002d4 R11: 0000000000e966e9 R12: ff507ec800013d10
R13: ff3cda1511636240 R14: 0000000000000000 R15: ff3cda1506dc6600
? srso_alias_return_thunk+0x5/0xfbef5
ep_send_events+0x182/0x2c0
ep_poll+0x13a/0x3c0
? __pfx_autoremove_wake_function+0x10/0x10
do_epoll_wait+0xbb/0xe0
? do_syscall_64+0x6b/0xe0
__x64_sys_epoll_wait+0x60/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
do_syscall_64+0x5c/0xe0
? handle_mm_fault+0x138/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x35d/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x61/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f643f50fb3e
Code: Unable to access opcode bytes at 0x7f643f50fb14.
RSP: 002b:00007ffee93e8ec0 EFLAGS: 00000293 ORIG_RAX: 00000000000000e8
RAX: ffffffffffffffda RBX: 000055b1acbfff40 RCX: 00007f643f50fb3e
RDX: 00000000000000b2 RSI: 000055b1acd93370 RDI: 0000000000000004
RBP: 000055b1acc000d0 R08: 0000000000000000 R09: 23ff68377f3a9649
R10: 00000000ffffffff R11: 0000000000000293 R12: 0000000000000302
R13: 000055b1acbfff40 R14: 00000000000000b2 R15: 000000000000004d
</TASK>
Mem-Info:
active_anon:0 inactive_anon:0 isolated_anon:0
Node 0 active_anon:0kB inactive_anon:0kB active_file:1142680kB inactive_file:2343368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4kB dirty:276388kB writeback:59904kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:3104kB pagetables:2824kB sec_pagetables:0kB all_unreclaimable? yes
Node 0 DMA free:14668kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:180kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1678 3599 3599 3599
Node 0 DMA32 free:38936kB boost:0kB min:31392kB low:39240kB high:47088kB reserved_highatomic:4096KB active_anon:0kB inactive_anon:0kB active_file:531376kB inactive_file:1143480kB unevictable:0kB writepending:328168kB present:2080624kB managed:1752944kB mlocked:0kB bounce:0kB free_pcp:192kB local_pcp:192kB free_cma:0kB
lowmem_reserve[]: 0 0 1920 1920 1920
Node 0 Normal free:35368kB boost:10240kB min:46148kB low:55124kB high:64100kB reserved_highatomic:2048KB active_anon:0kB inactive_anon:0kB active_file:611052kB inactive_file:1199960kB unevictable:0kB writepending:8124kB present:2097152kB managed:1972736kB mlocked:0kB bounce:0kB free_pcp:2020kB local_pcp:1224kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 1*4kB (M) 1*8kB (M) 0*16kB 0*32kB 1*64kB (M) 0*128kB 1*256kB (M) 2*512kB (UM) 1*1024kB (M) 2*2048kB (U) 2*4096kB (M) = 14668kB
Node 0 DMA32: 76*4kB (UM) 13*8kB (UME) 16*16kB (UME) 20*32kB (UME) 4*64kB (U) 2*128kB (UM) 3*256kB (UM) 1*512kB (U) 17*1024kB (UM) 7*2048kB (UM) 1*4096kB (M) = 38936kB
Node 0 Normal: 854*4kB (UME) 1000*8kB (UME) 741*16kB (UME) 204*32kB (UME) 45*64kB (UM) 5*128kB (UM) 2*256kB (M) 1*512kB (U) 1*1024kB (U) 0*2048kB 0*4096kB = 35368kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
634651 total pagecache pages
0 pages in swap cache
Free swap = 3720036kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
113182 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 503] 0 503 7095 64 90112 320 -250 systemd-journal
[ 519] 0 519 9042 32 94208 544 -1000 systemd-udevd
[ 523] 32 523 3690 64 65536 224 0 rpcbind
[ 533] 0 533 4945 34 53248 224 -1000 auditd
[ 583] 81 583 2686 64 65536 128 -900 dbus-broker-lau
[ 584] 0 584 64978 32 143360 768 0 NetworkManager
[ 585] 81 585 1258 64 45056 96 -900 dbus-broker
[ 590] 0 590 20683 64 61440 64 0 irqbalance
[ 592] 0 592 7232 32 81920 320 0 systemd-logind
[ 597] 997 597 21229 32 69632 160 0 chronyd
[ 619] 0 619 4378 64 73728 352 -1000 sshd
[ 620] 0 620 20265 32 61440 32 0 rhsmcertd
[ 624] 996 624 52786 37 81920 352 0 munged
[ 628] 0 628 14150 49 86016 224 0 gssproxy
[ 659] 0 659 41190 96 81920 544 0 rsyslogd
[ 665] 29 665 2839 75 57344 192 0 rpc.statd
[ 723] 0 723 85060 52 126976 512 0 automount
[ 732] 0 732 764 32 45056 32 0 agetty
[ 733] 0 733 1406 64 49152 0 0 agetty
[ 748] 0 748 2160 64 61440 192 0 crond
[ 973] 0 973 5212 64 86016 512 0 sshd
[ 977] 0 977 5578 64 77824 448 100 systemd
[ 979] 0 979 43187 59 106496 864 100 (sd-pam)
[ 986] 0 986 5212 66 86016 512 0 sshd
[ 1373] 0 1373 1784 64 53248 32 0 run_test_wrappe
[ 1416] 0 1416 1817 32 49152 64 0 run_test.sh
[ 1738] 0 1738 3463 70 73728 1632 0 bash
[ 117582] 0 117582 3463 55 69632 1600 0 bash
[ 117583] 0 117583 757 32 40960 0 0 tee
[ 117791] 0 117791 2759 49 61440 1632 0 bash
[ 121948] 0 121948 5212 64 77824 480 0 sshd
[ 121957] 0 121957 3323 61 69632 352 0 ssh
[ 121958] 0 121958 3339 60 61440 384 0 ssh
[ 121963] 0 121963 3323 60 69632 352 0 ssh
[ 121964] 0 121964 3323 61 69632 384 0 ssh
[ 121965] 0 121965 5212 75 77824 512 0 sshd
[ 123215] 0 123215 2759 58 61440 1664 0 bash
[ 123216] 0 123216 757 64 45056 0 0 tee
[ 125504] 500 125504 10871 47 90112 4640 0 iozone
[ 125505] 0 125505 757 32 45056 0 0 tee
[ 125527] 500 125527 10871 64 90112 4621 0 iozone
[ 125528] 500 125528 10871 64 90112 4621 0 iozone
[ 125529] 500 125529 10871 64 90112 4621 0 iozone
[ 125530] 500 125530 10871 64 90112 4653 0 iozone