| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| parallel-scale test statahead: statahead test, multiple clients | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 1 UID: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd out_of_memory.cold+0x61/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0x8d/0x1e0 folio_alloc_mpol_noprof+0x14/0x30 __read_swap_cache_async+0x1ac/0x250 swap_cluster_readahead+0x234/0x2f0 ? pick_next_task_fair+0x14a/0x400 swapin_readahead+0x6d/0xe0 do_swap_page+0x29f/0x1190 ? __mod_memcg_lruvec_state+0xe4/0x1e0 ? __pfx_default_wake_function+0x10/0x10 __handle_mm_fault+0x30b/0x740 handle_mm_fault+0x115/0x370 do_user_addr_fault+0x158/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0010:__get_user_8+0x1a/0x30 | Lustre: lustre-OST0000-osc-ffff8c1a82bc1000: disconnect after 24s idle Lustre: Skipped 7 previous similar messages obd_memory max: 479030133, obd_memory current: 10292680 systemd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 CPU: 1 UID: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 dump_header+0x43/0x1ad out_of_memory.cold+0x3e/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0x8d/0x1e0 folio_alloc_mpol_noprof+0x14/0x30 __read_swap_cache_async+0x1ac/0x250 swap_cluster_readahead+0x234/0x2f0 ? pick_next_task_fair+0x14a/0x400 swapin_readahead+0x6d/0xe0 do_swap_page+0x29f/0x1190 ? __mod_memcg_lruvec_state+0xe4/0x1e0 ? __pfx_default_wake_function+0x10/0x10 __handle_mm_fault+0x30b/0x740 handle_mm_fault+0x115/0x370 do_user_addr_fault+0x158/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0010:__get_user_8+0x1a/0x30 Code: 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 f3 0f 1e fa 48 ba 00 f0 ff ff ff 7f 00 00 48 39 c2 48 19 d2 48 09 d0 0f 01 cb <48> 8b 10 31 c0 0f 01 ca e9 59 74 01 00 66 0f 1f 84 00 00 00 00 00 RSP: 0000:ffffd02300013e70 EFLAGS: 00050202 RAX: 00007f30b05932a8 RBX: ffff8c1a8027d100 RCX: 0000000000000001 RDX: 0000000000000000 RSI: ffffd02300013ea0 RDI: ffff8c1a8027d100 RBP: ffffd02300013f00 R08: 0000000000000002 R09: ffff8c1a88d3a7f0 R10: 0000000000000000 R11: ffff8c1a811ed60c R12: 00007f30b068b5ea R13: ffff8c1a8027d100 R14: ffffd02300013f48 R15: 0000000000000000 rseq_get_rseq_cs+0x1d/0x220 rseq_ip_fixup+0x66/0x1d0 __rseq_handle_notify_resume+0x29/0x60 irqentry_exit_to_user_mode+0x152/0x180 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7f30b068b5ea Code: Unable to access opcode bytes at 0x7f30b068b5c0. RSP: 002b:00007ffc009d0c60 EFLAGS: 00010202 RAX: 0000000000000002 RBX: 00007ffc009d1180 RCX: 0000000000000032 RDX: 0000000000000000 RSI: 00007f30b0b65770 RDI: 00007ffc009d1180 RBP: 0000000000000000 R08: 0000000000000069 R09: 00000000fffffffe R10: 0000000000000000 R11: 0000000000000000 R12: 00007f30b0b65770 R13: 0000000000000000 R14: 00007ffc009d1240 R15: 00007f30b0b65770 </TASK> Mem-Info: active_anon:461132 inactive_anon:384299 isolated_anon:0 Node 0 active_anon:1894424kB inactive_anon:1487300kB active_file:104kB inactive_file:92kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:12kB dirty:0kB writeback:0kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2240512kB writeback_tmp:0kB kernel_stack:5632kB pagetables:18828kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB Node 0 DMA free:14812kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:48kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1664 3638 3638 3638 Node 0 DMA32 free:37324kB boost:0kB min:30308kB low:37884kB high:45460kB reserved_highatomic:0KB active_anon:1123176kB inactive_anon:539748kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2080624kB managed:1704056kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1974 1974 1974 Node 0 Normal free:36916kB boost:83232kB min:120224kB low:129472kB high:138720kB reserved_highatomic:0KB active_anon:905920kB inactive_anon:812588kB active_file:104kB inactive_file:92kB unevictable:0kB writepending:0kB present:2097152kB managed:2021608kB mlocked:0kB bounce:0kB free_pcp:248kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 1*4kB (M) 1*8kB (M) 1*16kB (M) 0*32kB 1*64kB (M) 1*128kB (M) 1*256kB (M) 2*512kB (UM) 1*1024kB (M) 0*2048kB 3*4096kB (M) = 14812kB Node 0 DMA32: 13*4kB (M) 5*8kB (M) 17*16kB (M) 42*32kB (M) 39*64kB (M) 37*128kB (M) 23*256kB (M) 18*512kB (M) 13*1024kB (M) 0*2048kB 0*4096kB = 37356kB Node 0 Normal: 9229*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36916kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 252 total pagecache pages 222 pages in swap cache Free swap = 1892692kB Total swap = 2841596kB 1048442 pages RAM 0 pages HighMem/MovableOnly 113186 pages reserved 0 pages cma reserved 0 pages hwpoisoned Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 619] 0 619 10411 38 0 31 7 86016 320 -250 systemd-journal [ 648] 0 648 3972 88 32 56 0 73728 224 0 systemd-userdbd [ 666] 0 666 8534 42 32 10 0 94208 576 -1000 systemd-udevd [ 710] 32 710 3163 0 0 0 0 65536 224 0 rpcbind [ 711] 0 711 5103 46 14 32 0 53248 256 -1000 auditd [ 756] 81 756 2212 44 32 12 0 57344 160 -900 dbus-broker-lau [ 757] 81 757 1225 47 0 47 0 49152 128 -900 dbus-broker [ 758] 0 758 82594 10 0 10 0 135168 736 0 NetworkManager [ 761] 0 761 20104 17 0 17 0 61440 64 0 irqbalance [ 762] 0 762 4184 10 0 10 0 77824 288 0 systemd-logind [ 773] 994 773 21121 31 10 21 0 73728 128 0 chronyd [ 803] 0 803 30676 29 0 29 0 94208 576 0 gssproxy [ 805] 0 805 2196 35 0 35 0 65536 288 -1000 sshd [ 811] 0 811 20279 22 0 22 0 53248 32 0 rhsmcertd [ 873] 0 873 42803 19 0 0 19 90112 224 0 rsyslogd [ 877] 29 877 2654 51 11 40 0 61440 192 0 rpc.statd [ 882] 996 882 51989 52 40 12 0 69632 416 0 munged [ 942] 0 942 84321 38 0 38 0 122880 480 0 automount [ 953] 0 953 1246 25 0 25 0 49152 32 0 agetty [ 955] 0 955 1888 42 0 42 0 57344 32 0 agetty [ 982] 0 982 2075 45 0 45 0 53248 224 0 crond [ 5151] 0 5151 3878 52 32 20 0 73728 480 0 sshd-session [ 5156] 0 5156 5132 27 0 27 0 81920 480 100 systemd [ 5158] 0 5158 3415 80 30 50 0 53248 288 100 (sd-pam) [ 5690] 0 5690 1738 65 32 33 0 49152 64 0 run_test_wrappe [ 5734] 0 5734 1771 20 0 20 0 49152 64 0 run_test.sh [ 6085] 0 6085 3685 54 37 17 0 69632 1888 0 bash [ 761272] 0 761272 3685 55 37 18 0 65536 1888 0 bash [ 761273] 0 761273 709 49 32 17 0 45056 0 0 tee [ 761476] 0 761476 2976 52 1 51 0 53248 1920 0 bash [ 778656] 0 778656 2976 54 33 21 0 53248 1920 0 bash [ 778657] 0 778657 710 32 32 0 0 49152 0 0 tee [ 778887] 0 778887 2976 60 33 27 0 53248 1920 0 bash [ 778888] 0 778888 710 40 0 40 0 45056 32 0 tee [ 778889] 0 778889 1785 73 32 41 0 53248 160 0 su [ 778890] 532 778890 189988 94 40 54 0 167936 1152 0 prte [ 778893] 532 778893 3235 60 32 28 0 73728 448 0 ssh [ 778902] 532 778902 48510 12882 12827 55 0 266240 5056 0 mdsrate [ 778903] 532 778903 48511 14050 14012 38 0 253952 3872 0 mdsrate [ 778904] 532 778904 48472 12851 12818 33 0 253952 5056 0 mdsrate [ 778905] 532 778905 48511 14463 14430 33 0 253952 3456 0 mdsrate [ 778906] 532 778906 48472 12510 12510 0 0 225280 1152 0 mdsrate [ 778907] 532 778907 48511 12541 12540 1 0 253952 4672 0 mdsrate [ 778908] 532 778908 48511 16074 16029 45 0 266240 1856 0 mdsrate [ 778909] 532 778909 48472 12523 12506 17 0 241664 2656 0 mdsrate [ 778910] 532 778910 44330 12516 12506 10 0 233472 1152 0 mdsrate [ 778911] 532 778911 48511 12744 12689 55 0 249856 5152 0 mdsrate [ 778912] 532 778912 48511 14208 14173 35 0 262144 3744 0 mdsrate [ 778913] 532 778913 44291 12553 12537 16 0 225280 1152 0 mdsrate [ 778914] 532 778914 48472 12516 12510 6 0 258048 5344 0 mdsrate [ 778915] 532 778915 48511 13117 13084 33 0 262144 4800 0 mdsrate [ 778916] 532 778916 44291 12566 12530 36 0 233472 1152 0 mdsrate [ 778917] 532 778917 48512 13735 13681 54 0 266240 4192 0 mdsrate [ 778918] 532 778918 48472 12805 12776 29 0 270336 5120 0 mdsrate [ 778919] 532 778919 44291 12539 12534 5 0 225280 1152 0 mdsrate [ 778920] 532 778920 44291 12512 12510 2 0 225280 1152 0 mdsrate [ 778921] 532 778921 48472 12960 12925 35 0 258048 4960 0 mdsrate [ 778922] 532 778922 48511 14123 14101 22 0 258048 3744 0 mdsrate [ 778923] 532 778923 48511 13437 13384 53 0 253952 4512 0 mdsrate [ 778924] 532 778924 48511 13866 13833 33 0 266240 4064 0 mdsrate [ 778925] 532 778925 48511 13884 13844 40 0 262144 4032 0 mdsrate [ 778926] 532 778926 48511 12670 12626 44 0 262144 5248 0 mdsrate [ 778927] 532 778927 48511 13102 13055 47 0 249856 4832 0 mdsrate [ 778928] 532 778928 48511 14299 14261 38 0 262144 3616 0 mdsrate [ 778929] 532 778929 48511 13092 13049 43 0 270336 4864 0 mdsrate [ 778930] 532 778930 48511 13617 13569 48 0 262144 4320 0 mdsrate [ 778931] 532 778931 48511 12616 12572 44 0 258048 5312 0 mdsrate [ 778932] 532 778932 44291 12553 12517 36 0 229376 1152 0 mdsrate [ 778933] 532 778933 48512 14424 14397 27 0 258048 3488 0 mdsrate [ 778934] 532 778934 48511 12748 12735 13 0 266240 5120 0 mdsrate [ 778935] 532 778935 48472 13080 13022 58 0 253952 4832 0 mdsrate [ 778936] 532 778936 48512 14671 14652 19 0 258048 3232 0 mdsrate [ 778937] 532 778937 48512 13654 13629 25 0 249856 4256 0 mdsrate [ 778938] 532 778938 44290 12540 12510 30 0 221184 1184 0 mdsrate [ 778939] 532 778939 44290 12528 12507 21 0 229376 1152 0 mdsrate [ 778940] 532 778940 48511 13476 13469 7 0 262144 4416 0 mdsrate [ 778941] 532 778941 44290 12481 12481 0 0 229376 1152 0 mdsrate [ 778942] 532 778942 44330 12527 12509 18 0 233472 1184 0 mdsrate [ 778943] 532 778943 48511 13929 13886 43 0 253952 4000 0 mdsrate [ 778944] 532 778944 48511 13238 13212 26 0 262144 4672 0 mdsrate [ 778945] 532 778945 48511 12988 12952 36 0 262144 4896 0 mdsrate [ 778946] 532 778946 44291 12569 12542 27 0 225280 1184 0 mdsrate [ 778947] 532 778947 44290 12539 12528 11 0 229376 1152 0 mdsrate [ 778948] 532 778948 44290 12524 12489 35 0 225280 1152 0 mdsrate [ 778949] 532 778949 48511 13026 12989 37 0 262144 4896 0 mdsrate [ 778950] 532 778950 48512 16713 16701 12 0 266240 1184 0 mdsrate [ 778951] 532 778951 48511 12933 12925 8 0 262144 4928 0 mdsrate [ 778953] 532 778953 48512 16767 16721 46 0 262144 1184 0 mdsrate [ 778954] 532 778954 48511 13760 13724 36 0 258048 4160 0 mdsrate [ 778956] 532 778956 48511 14052 14022 30 0 253952 3840 0 mdsrate [ 778957] 532 778957 48511 13891 13854 37 0 253952 4064 0 mdsrate [ 778958] 532 778958 48511 15294 15273 21 0 262144 2592 0 mdsrate [ 778959] 532 778959 48511 13520 13502 18 0 258048 4352 0 mdsrate [ 778960] 532 778960 48511 12554 12543 11 0 258048 2464 0 mdsrate [ 778965] 532 778965 48511 13150 13116 34 0 266240 4768 0 mdsrate [ 778971] 532 778971 44330 12530 12530 0 0 225280 1184 0 mdsrate [ 778978] 532 778978 48511 13723 13701 22 0 253952 4160 0 mdsrate [ 778989] 532 778989 48511 12574 12545 29 0 262144 5312 0 mdsrate [ 778993] 532 778993 48511 12677 12659 18 0 258048 5184 0 mdsrate [ 778994] 532 778994 44291 12548 12528 20 0 229376 1152 0 mdsrate [ 779024] 532 779024 48511 12783 12755 28 0 258048 5120 0 mdsrate [ 779118] 0 779118 4083 44 0 44 0 73728 256 0 systemd-userwor [ 779119] 0 779119 3948 78 32 46 0 57344 32 0 systemd-userwor [ 779121] 0 779121 1947 0 0 0 0 49152 32 0 systemd-userwor [ 779122] 0 779122 2196 30 0 30 0 57344 283 0 sshd | Link to test (2026-10-22 10:11) |