| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanity-benchmark test iozone: iozone | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 0 PID: 122577 Comm: iozone Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.60.1.el9_6.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb out_of_memory.cold+0x5e/0x7e __alloc_pages_slowpath.constprop.0+0x798/0xb20 __alloc_pages+0x21d/0x250 alloc_pages_mpol+0x93/0x1e0 __read_swap_cache_async+0x12e/0x280 swap_cluster_readahead+0x14f/0x2c0 swapin_readahead+0xce/0xe0 do_swap_page+0x41a/0xa50 ? __pte_offset_map+0x17/0x140 __handle_mm_fault+0x2fe/0x650 handle_mm_fault+0xe9/0x250 do_user_addr_fault+0x154/0x620 exc_page_fault+0x62/0x150 asm_exc_page_fault+0x22/0x30 RIP: 0033:0x421097 | Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 7244016kB available, using 7486816kB file size Lustre: DEBUG MARKER: min OST has 7244016kB available, using 7486816kB file size Lustre: 3050:0:(client.c:2289:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1792752756/real 1792752756] req@00000000e7b94621 x1879822617030080/t0(0) o4->lustre-OST0005-osc-ffff9a26446ea800@10.240.30.216@tcp:6/4 lens 488/448 e 0 to 1 dl 1792752807 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'iozone.500' Lustre: 3050:0:(client.c:2289:ptlrpc_expire_one_request()) Skipped 5 previous similar messages Lustre: lustre-OST0005-osc-ffff9a26446ea800: Connection to lustre-OST0005 (at 10.240.30.216@tcp) was lost; in progress operations using this service will wait for recovery to complete obd_memory max: 2968653675, obd_memory current: 52454755 iozone invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 CPU: 0 PID: 122577 Comm: iozone Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.60.1.el9_6.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 dump_header+0x4a/0x213 out_of_memory.cold+0x39/0x7e __alloc_pages_slowpath.constprop.0+0x798/0xb20 __alloc_pages+0x21d/0x250 alloc_pages_mpol+0x93/0x1e0 __read_swap_cache_async+0x12e/0x280 swap_cluster_readahead+0x14f/0x2c0 swapin_readahead+0xce/0xe0 do_swap_page+0x41a/0xa50 ? __pte_offset_map+0x17/0x140 __handle_mm_fault+0x2fe/0x650 handle_mm_fault+0xe9/0x250 do_user_addr_fault+0x154/0x620 exc_page_fault+0x62/0x150 asm_exc_page_fault+0x22/0x30 RIP: 0033:0x421097 Code: Unable to access opcode bytes at 0x42106d. RSP: 002b:00007ffcd9323980 EFLAGS: 00010286 RAX: 0000000000004000 RBX: 0000000000080000 RCX: e403e605e807e47d RDX: 00007ff3cc002008 RSI: 00007ff3cc002020 RDI: e302e504e706e37c RBP: 00007ff3cc000000 R08: b8b8b8b8b8b8b8b8 R09: 0101010101010101 R10: 0000000000000020 R11: 0000000000000100 R12: 00000000b8b8b8b8 R13: 0000000000000000 R14: 0000000000000003 R15: 28f5c28f5c28f5c3 </TASK> Mem-Info: active_anon:0 inactive_anon:7 isolated_anon:0 Node 0 active_anon:0kB inactive_anon:28kB active_file:1090252kB inactive_file:2369872kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4kB dirty:191488kB writeback:262160kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:3312kB pagetables:2580kB sec_pagetables:0kB all_unreclaimable? yes Node 0 DMA free:14680kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:72kB unevictable:0kB writepending:72kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1678 3600 3600 3600 Node 0 DMA32 free:40504kB boost:0kB min:31372kB low:39212kB high:47052kB reserved_highatomic:2048KB active_anon:0kB inactive_anon:24kB active_file:529696kB inactive_file:1140904kB unevictable:0kB writepending:431136kB present:2080624kB managed:1752944kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1922 1922 1922 Node 0 Normal free:35532kB boost:0kB min:35928kB low:44908kB high:53888kB reserved_highatomic:0KB active_anon:0kB inactive_anon:4kB active_file:562236kB inactive_file:1227468kB unevictable:0kB writepending:22440kB present:2097152kB managed:1975104kB mlocked:0kB bounce:0kB free_pcp:1440kB local_pcp:388kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 2*4kB (M) 2*8kB (M) 0*16kB 2*32kB (UM) 0*64kB 2*128kB (UM) 2*256kB (UM) 1*512kB (M) 1*1024kB (M) 0*2048kB 3*4096kB (ME) = 14680kB Node 0 DMA32: 250*4kB (UME) 276*8kB (UME) 243*16kB (UME) 208*32kB (UME) 156*64kB (UME) 83*128kB (UME) 18*256kB (UME) 3*512kB (UME) 0*1024kB 0*2048kB 0*4096kB = 40504kB Node 0 Normal: 497*4kB (UME) 351*8kB (UME) 91*16kB (UME) 53*32kB (UME) 55*64kB (UME) 82*128kB (UM) 31*256kB (UM) 7*512kB (UM) 0*1024kB 1*2048kB (U) 0*4096kB = 35532kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 737802 total pagecache pages 7 pages in swap cache Free swap = 2748516kB Total swap = 2843644kB 1048442 pages RAM 0 pages HighMem/MovableOnly 112590 pages reserved 0 pages cma reserved 0 pages hwpoisoned Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 507] 0 507 6922 64 81920 320 -250 systemd-journal [ 526] 0 526 8703 27 86016 448 -1000 systemd-udevd [ 552] 32 552 3511 32 69632 192 0 rpcbind [ 554] 0 554 4771 54 61440 704 -1000 auditd [ 606] 81 606 2680 32 57344 160 -900 dbus-broker-lau [ 607] 81 607 1253 32 49152 96 -900 dbus-broker [ 608] 0 608 64801 66 143360 1280 0 NetworkManager [ 611] 0 611 20674 32 61440 96 0 irqbalance [ 612] 0 612 7055 64 77824 288 0 systemd-logind [ 619] 997 619 21224 32 65536 160 0 chronyd [ 644] 0 644 20263 76 61440 512 0 rhsmcertd [ 645] 0 645 4165 64 73728 320 -1000 sshd [ 650] 996 650 52614 41 77824 320 0 munged [ 651] 0 651 13968 48 86016 736 0 gssproxy [ 697] 0 697 41183 32 81920 960 0 rsyslogd [ 698] 29 698 2665 41 53248 160 0 rpc.statd [ 736] 0 736 84863 49 122880 992 0 automount [ 740] 0 740 761 32 40960 32 0 agetty [ 744] 0 744 1403 32 49152 32 0 agetty [ 778] 0 778 2157 32 61440 224 0 crond [ 979] 0 979 4996 32 77824 480 0 sshd [ 983] 0 983 5401 64 81920 416 100 systemd [ 985] 0 985 26622 64 86016 832 100 (sd-pam) [ 992] 0 992 4996 57 77824 448 0 sshd [ 1348] 0 1348 1781 32 61440 64 0 run_test_wrappe [ 1387] 0 1387 1814 32 57344 64 0 run_test.sh [ 1688] 0 1688 3460 38 69632 1600 0 bash [ 115849] 0 115849 3460 44 61440 1632 0 bash [ 115850] 0 115850 754 32 45056 32 0 tee [ 116050] 0 116050 2755 49 69632 1664 0 bash [ 120294] 0 120294 4997 64 77824 448 0 sshd [ 120297] 0 120297 19568 32 151552 640 0 sssd_kcm [ 120301] 0 120301 3122 56 61440 320 0 ssh [ 120305] 0 120305 3113 50 65536 288 0 ssh [ 120309] 0 120309 3082 32 61440 320 0 ssh [ 120311] 0 120311 3116 32 65536 320 0 ssh [ 120312] 0 120312 4997 64 73728 448 0 sshd [ 121566] 0 121566 2755 58 69632 1658 0 bash [ 121567] 0 121567 754 32 45056 0 0 tee [ 122577] 500 122577 10856 47 90112 4640 0 iozone [ 122578] 0 122578 754 32 45056 0 0 tee | Link to test (2026-10-23 13:22) |