| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| parallel-scale test statahead: statahead test, multiple clients | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 1 UID: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd out_of_memory.cold+0x61/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0x8d/0x1e0 folio_alloc_noprof+0x5b/0xa0 __filemap_get_folio+0x19f/0x2f0 filemap_fault+0x1fb/0x9b0 __do_fault+0x33/0x180 do_read_fault+0x11e/0x1f0 do_fault+0x1fd/0x330 __handle_mm_fault+0x30b/0x740 handle_mm_fault+0x115/0x370 do_user_addr_fault+0x20c/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7fa228c7d2b0 | mdsrate: page allocation failure: order:0, mode:0x10dc0(GFP_KERNEL|__GFP_ZERO|__GFP_NORETRY), nodemask=(null),cpuset=/,mems_allowed=0 CPU: 0 UID: 532 PID: 455943 Comm: mdsrate Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 warn_alloc+0x164/0x190 __alloc_pages_slowpath.constprop.0+0x70b/0xb70 ? get_page_from_freelist+0x30e/0x5d0 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0x8d/0x1e0 get_free_pages_noprof+0x11/0x40 mlx5r_umr_alloc_xlt+0xc7/0x140 [mlx5_ib] ? try_to_wake_up+0x321/0x760 mlx5r_umr_create_xlt+0x4c/0x170 [mlx5_ib] _mlx5r_umr_update_mr_pas+0x108/0x4e0 [mlx5_ib] create_real_mr+0x195/0x1b0 [mlx5_ib] ? rdma_lookup_get_uobject+0xe9/0x160 [ib_uverbs] ib_uverbs_reg_mr+0x1ac/0x2e0 [ib_uverbs] ? __pfx_ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0x10/0x10 [ib_uverbs] ib_uverbs_handler_UVERBS_METHOD_INVOKE_WRITE+0xd4/0x140 [ib_uverbs] ib_uverbs_run_method+0x2f3/0x320 [ib_uverbs] ib_uverbs_cmd_verbs+0x223/0x2e0 [ib_uverbs] ? wake_all_kswapds+0x62/0xc0 ib_uverbs_ioctl+0xbb/0x130 [ib_uverbs] __x64_sys_ioctl+0x94/0xd0 do_syscall_64+0x7d/0x160 ? __count_memcg_events+0xdf/0x170 ? handle_mm_fault+0x256/0x370 ? do_user_addr_fault+0x347/0x640 ? exc_page_fault+0x73/0x160 entry_SYSCALL_64_after_hwframe+0x76/0x7e RIP: 0033:0x7f97e71284bf Code: 00 48 89 44 24 18 31 c0 48 8d 44 24 60 c7 04 24 10 00 00 00 48 89 44 24 08 48 8d 44 24 20 48 89 44 24 10 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 18 48 8b 44 24 18 64 48 2b 04 25 28 00 00 RSP: 002b:00007fff3ff13ba0 EFLAGS: 00000246 ORIG_RAX: 0000000000000010 RAX: ffffffffffffffda RBX: 00007fff3ff13cb0 RCX: 00007f97e71284bf RDX: 00007fff3ff13cd0 RSI: 00000000c0181b01 RDI: 0000000000000014 RBP: 00007fff3ff13ce8 R08: 000000003dfdd420 R09: 000000003dfdd2e0 R10: 0000000000000003 R11: 0000000000000246 R12: 00007fff3ff13ce8 R13: 00007fff3ff13e14 R14: 000000000000000c R15: 000000003dfdd420 </TASK> Mem-Info: active_anon:68160 inactive_anon:756626 isolated_anon:0 Node 0 active_anon:272640kB inactive_anon:3026504kB active_file:10432kB inactive_file:83380kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:53964kB dirty:36kB writeback:10836kB shmem:728kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2222080kB writeback_tmp:0kB kernel_stack:5808kB pagetables:17580kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB Node 0 DMA free:14828kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:36kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1664 3638 3638 3638 Node 0 DMA32 free:40064kB boost:2048kB min:32356kB low:39932kB high:47508kB reserved_highatomic:0KB active_anon:6700kB inactive_anon:1656224kB active_file:0kB inactive_file:8kB unevictable:0kB writepending:0kB present:2080624kB managed:1704056kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1974 1974 1974 Node 0 Normal free:52804kB boost:16384kB min:53376kB low:62624kB high:71872kB reserved_highatomic:0KB active_anon:265904kB inactive_anon:1370244kB active_file:11524kB inactive_file:81028kB unevictable:0kB writepending:10872kB present:2097152kB managed:2021608kB mlocked:0kB bounce:0kB free_pcp:768kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 1*4kB (M) 1*8kB (M) 1*16kB (M) 0*32kB 1*64kB (M) 1*128kB (M) 1*256kB (M) 2*512kB (UM) 1*1024kB (M) 0*2048kB 3*4096kB (M) = 14812kB Node 0 DMA32: 47*4kB (UM) 49*8kB (UM) 48*16kB (UM) 45*32kB (UM) 40*64kB (UM) 37*128kB (UM) 31*256kB (UM) 21*512kB (U) 11*1024kB (UM) 0*2048kB 0*4096kB = 40036kB Node 0 Normal: 2683*4kB (UME) 144*8kB (UME) 328*16kB (UME) 197*32kB (UME) 122*64kB (UME) 162*128kB (UME) 8*256kB (UM) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 54028kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 23761 total pagecache pages 567 pages in swap cache Free swap = 2833088kB Total swap = 2841596kB 1048442 pages RAM 0 pages HighMem/MovableOnly 113186 pages reserved 0 pages cma reserved 0 pages hwpoisoned Lustre: lustre-OST0000-osc-ffff8bef86a36800: disconnect after 20s idle Lustre: Skipped 6 previous similar messages obd_memory max: 630357521, obd_memory current: 10217641 systemd invoked oom-killer: gfp_mask=0x140cca(GFP_HIGHUSER_MOVABLE|__GFP_COMP), order=0, oom_score_adj=0 CPU: 1 UID: 0 PID: 1 Comm: systemd Kdump: loaded Tainted: G OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 dump_header+0x43/0x1ad out_of_memory.cold+0x3e/0x78 __alloc_pages_slowpath.constprop.0+0x594/0xb70 __alloc_pages_noprof+0x31b/0x330 alloc_pages_mpol_noprof+0x8d/0x1e0 folio_alloc_noprof+0x5b/0xa0 __filemap_get_folio+0x19f/0x2f0 filemap_fault+0x1fb/0x9b0 __do_fault+0x33/0x180 do_read_fault+0x11e/0x1f0 do_fault+0x1fd/0x330 __handle_mm_fault+0x30b/0x740 handle_mm_fault+0x115/0x370 do_user_addr_fault+0x20c/0x640 exc_page_fault+0x73/0x160 asm_exc_page_fault+0x26/0x30 RIP: 0033:0x7fa228c7d2b0 Code: Unable to access opcode bytes at 0x7fa228c7d286. RSP: 002b:00007ffda065e468 EFLAGS: 00010206 RAX: 0000564f3c55a9f0 RBX: 0000564f3c55a9f0 RCX: 0000000000000001 RDX: 0000000000000000 RSI: 0000564f3c55a9f0 RDI: 0000564f3c55a9f0 RBP: 0000564f3c55a9f0 R08: a8bb8c919462efad R09: b82ad4098b7469fe R10: b13ad3c4283f9a5e R11: 0000000000000206 R12: 0000564f3c3daf50 R13: 0000564f3c3daf50 R14: 0000564f3c3daf50 R15: 0000000000000000 </TASK> Mem-Info: active_anon:483817 inactive_anon:353544 isolated_anon:0 Node 0 active_anon:2200328kB inactive_anon:1151116kB active_file:100kB inactive_file:288kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:12kB dirty:0kB writeback:0kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:2220032kB writeback_tmp:0kB kernel_stack:5808kB pagetables:19852kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB Node 0 DMA free:14812kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:48kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1664 3638 3638 3638 Node 0 DMA32 free:37384kB boost:0kB min:30308kB low:37884kB high:45460kB reserved_highatomic:0KB active_anon:544736kB inactive_anon:1097888kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:2080624kB managed:1704056kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1974 1974 1974 Node 0 Normal free:36992kB boost:0kB min:36992kB low:46240kB high:55488kB reserved_highatomic:2048KB active_anon:378760kB inactive_anon:1324928kB active_file:0kB inactive_file:388kB unevictable:0kB writepending:0kB present:2097152kB managed:2021608kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 1*4kB (M) 1*8kB (M) 1*16kB (M) 0*32kB 1*64kB (M) 1*128kB (M) 1*256kB (M) 2*512kB (UM) 1*1024kB (M) 0*2048kB 3*4096kB (M) = 14812kB Node 0 DMA32: 186*4kB (UME) 146*8kB (M) 127*16kB (UME) 129*32kB (UME) 94*64kB (ME) 62*128kB (ME) 24*256kB (M) 12*512kB (M) 3*1024kB (UM) 0*2048kB 0*4096kB = 37384kB Node 0 Normal: 9268*4kB (M) 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37072kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 82 total pagecache pages 12 pages in swap cache Free swap = 1756276kB Total swap = 2841596kB 1048442 pages RAM 0 pages HighMem/MovableOnly 113186 pages reserved 0 pages cma reserved 0 pages hwpoisoned Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 621] 0 621 6137 33 32 1 0 77824 320 -250 systemd-journal [ 661] 0 661 3972 65 32 33 0 69632 256 0 systemd-userdbd [ 670] 0 670 8541 41 32 9 0 90112 576 -1000 systemd-udevd [ 693] 32 693 3163 0 0 0 0 65536 256 0 rpcbind [ 710] 0 710 5103 25 14 11 0 57344 256 -1000 auditd [ 736] 81 736 2212 5 0 5 0 61440 160 -900 dbus-broker-lau [ 763] 81 763 1225 41 0 41 0 53248 128 -900 dbus-broker [ 764] 0 764 82594 27 0 27 0 147456 736 0 NetworkManager [ 767] 0 767 20104 12 0 12 0 57344 96 0 irqbalance [ 768] 0 768 4184 34 32 2 0 77824 288 0 systemd-logind [ 780] 994 780 21121 8 8 0 0 73728 160 0 chronyd [ 808] 0 808 30676 12 0 12 0 90112 544 0 gssproxy [ 817] 0 817 2196 34 0 34 0 53248 256 -1000 sshd [ 818] 0 818 20279 39 0 39 0 61440 32 0 rhsmcertd [ 882] 0 882 40577 44 0 44 0 69632 256 0 rsyslogd [ 885] 29 885 2654 58 11 47 0 53248 160 0 rpc.statd [ 888] 996 888 51989 8 8 0 0 77824 448 0 munged [ 945] 0 945 84317 55 32 23 0 122880 480 0 automount [ 951] 0 951 1246 9 0 9 0 57344 32 0 agetty [ 953] 0 953 1888 51 0 51 0 53248 32 0 agetty [ 1035] 0 1035 2075 32 32 0 0 65536 192 0 crond [ 1176] 0 1176 3790 52 32 20 0 73728 448 0 sshd-session [ 1181] 0 1181 5135 61 32 29 0 86016 448 100 systemd [ 1183] 0 1183 3415 38 16 22 0 65536 288 100 (sd-pam) [ 1192] 0 1192 3851 47 1 46 0 73728 557 0 sshd-session [ 1595] 0 1595 1738 38 0 38 0 53248 64 0 run_test_wrappe [ 1639] 0 1639 1772 16 0 16 0 57344 64 0 run_test.sh [ 1982] 0 1982 3737 66 35 31 0 77824 1952 0 bash [ 437532] 0 437532 3737 59 23 36 0 73728 1952 0 bash [ 437533] 0 437533 709 6 0 6 0 49152 32 0 tee [ 437727] 0 437727 2999 71 44 27 0 61440 1952 0 bash [ 450867] 0 450867 3162 31 5 26 0 65536 416 0 ssh [ 450870] 0 450870 3162 24 5 19 0 61440 391 0 ssh [ 450873] 0 450873 3167 56 7 49 0 69632 391 0 ssh [ 450877] 0 450877 3167 7 5 2 0 65536 384 0 ssh [ 455653] 0 455653 2999 77 53 24 0 61440 1920 0 bash [ 455654] 0 455654 709 51 32 19 0 53248 0 0 tee [ 455867] 0 455867 2999 102 53 49 0 61440 1920 0 bash [ 455868] 0 455868 709 19 0 19 0 49152 32 0 tee [ 455869] 0 455869 1784 48 32 16 0 53248 128 0 su [ 455871] 532 455871 189985 54 21 33 0 172032 1152 0 prte [ 455874] 532 455874 3233 14 0 14 0 65536 448 0 ssh [ 455883] 532 455883 48513 16738 16701 37 0 262144 1216 0 mdsrate [ 455884] 532 455884 48515 16768 16738 30 0 253952 1152 0 mdsrate [ 455885] 532 455885 48474 12733 12709 24 0 258048 5152 0 mdsrate [ 455886] 532 455886 44294 12525 12505 20 0 225280 1152 0 mdsrate [ 455887] 532 455887 48514 13119 13102 17 0 258048 4768 0 mdsrate [ 455888] 532 455888 48513 15191 15131 60 0 266240 2720 0 mdsrate [ 455889] 532 455889 48514 16687 16687 0 0 262144 1216 0 mdsrate [ 455890] 532 455890 48521 12884 12825 59 0 258048 5056 0 mdsrate [ 455891] 532 455891 48513 12918 12874 44 0 262144 4992 0 mdsrate [ 455892] 532 455892 48514 13217 13185 32 0 258048 4704 0 mdsrate [ 455893] 532 455893 48513 13411 13384 27 0 262144 4480 0 mdsrate [ 455894] 532 455894 48513 14367 14321 46 0 262144 3552 0 mdsrate [ 455895] 532 455895 48514 13341 13301 40 0 266240 4576 0 mdsrate [ 455896] 532 455896 48514 13004 12969 35 0 258048 4928 0 mdsrate [ 455897] 532 455897 48513 13057 13040 17 0 262144 4832 0 mdsrate [ 455898] 532 455898 48514 12650 12628 22 0 270336 5248 0 mdsrate [ 455899] 532 455899 48474 12578 12545 33 0 262144 5312 0 mdsrate [ 455900] 532 455900 48513 13151 13113 38 0 258048 4800 0 mdsrate [ 455901] 532 455901 48513 13075 13019 56 0 262144 4832 0 mdsrate [ 455902] 532 455902 48474 13120 13112 8 0 262144 4736 0 mdsrate [ 455903] 532 455903 48475 12919 12910 9 0 258048 4960 0 mdsrate [ 455904] 532 455904 48474 12551 12539 12 0 253952 5344 0 mdsrate [ 455905] 532 455905 44332 12523 12502 21 0 229376 1184 0 mdsrate [ 455906] 532 455906 48475 12767 12732 35 0 266240 5120 0 mdsrate [ 455907] 532 455907 48514 13265 13234 31 0 266240 4640 0 mdsrate [ 455908] 532 455908 48474 12949 12906 43 0 258048 4928 0 mdsrate [ 455909] 532 455909 48475 14039 14012 27 0 266240 3840 0 mdsrate [ 455910] 532 455910 44293 12528 12528 0 0 229376 1152 0 mdsrate [ 455911] 532 455911 48514 13005 12978 27 0 258048 4896 0 mdsrate [ 455912] 532 455912 48513 15900 15881 19 0 258048 2016 0 mdsrate [ 455913] 532 455913 48474 13073 13042 31 0 262144 4832 0 mdsrate [ 455914] 532 455914 44332 12539 12509 30 0 225280 1184 0 mdsrate [ 455915] 532 455915 48514 12745 12733 12 0 262144 5152 0 mdsrate [ 455916] 532 455916 48514 12826 12816 10 0 266240 5056 0 mdsrate [ 455917] 532 455917 44332 12556 12530 26 0 229376 1152 0 mdsrate [ 455918] 532 455918 44294 12550 12512 38 0 229376 1152 0 mdsrate [ 455919] 532 455919 44293 12506 12495 11 0 225280 1184 0 mdsrate [ 455920] 532 455920 48474 12523 12495 28 0 258048 5344 0 mdsrate [ 455921] 532 455921 48513 12854 12820 34 0 258048 5056 0 mdsrate [ 455922] 532 455922 48514 12944 12903 41 0 266240 4992 0 mdsrate [ 455923] 532 455923 48514 12577 12541 36 0 249856 5312 0 mdsrate [ 455924] 532 455924 44294 12505 12505 0 0 225280 1152 0 mdsrate [ 455925] 532 455925 48513 14246 14209 37 0 253952 3648 0 mdsrate [ 455926] 532 455926 48474 13329 13300 29 0 258048 4576 0 mdsrate [ 455927] 532 455927 48513 12966 12905 61 0 262144 4960 0 mdsrate [ 455928] 532 455928 48474 13065 13036 29 0 262144 4800 0 mdsrate [ 455929] 532 455929 48513 13295 13295 0 0 258048 4576 0 mdsrate [ 455930] 532 455930 48513 12723 12719 4 0 253952 5120 0 mdsrate [ 455931] 532 455931 48513 14392 14369 23 0 262144 3520 0 mdsrate [ 455932] 532 455932 44293 12552 12533 19 0 221184 1152 0 mdsrate [ 455933] 532 455933 48474 15923 15911 12 0 258048 1952 0 mdsrate [ 455934] 532 455934 48514 14209 14177 32 0 262144 3712 0 mdsrate [ 455935] 532 455935 48474 12937 12937 0 0 258048 4928 0 mdsrate [ 455936] 532 455936 48513 12571 12570 1 0 258048 4128 0 mdsrate [ 455937] 532 455937 48514 13424 13395 29 0 258048 4480 0 mdsrate [ 455938] 532 455938 48474 12740 12715 25 0 262144 5120 0 mdsrate [ 455939] 532 455939 48474 12584 12571 13 0 258048 5280 0 mdsrate [ 455940] 532 455940 48475 12570 12546 24 0 253952 4000 0 mdsrate [ 455941] 532 455941 48514 13059 13008 51 0 258048 4864 0 mdsrate [ 455942] 532 455942 48513 12564 12544 20 0 262144 5344 0 mdsrate [ 455943] 532 455943 48474 12956 12939 17 0 253952 4928 0 mdsrate [ 455944] 532 455944 48513 13402 13365 37 0 258048 4544 0 mdsrate [ 455945] 532 455945 48474 13041 13014 27 0 258048 4864 0 mdsrate [ 455946] 532 455946 48513 15307 15281 26 0 258048 2624 0 mdsrate [ 456086] 0 456086 1771 18 0 18 0 53248 96 0 bash [ 456133] 0 456133 1000 57 32 25 0 45056 64 0 bash [ 456151] 0 456151 1000 50 0 50 0 53248 64 0 bash [ 456169] 0 456169 2242 89 32 57 0 61440 214 0 crond [ 456171] 0 456171 444 0 0 0 0 32768 0 0 bash [ 456173] 0 456173 1000 18 0 18 0 45056 76 0 bash [ 456174] 0 456174 4082 2 0 2 0 69632 224 0 systemd-userwor [ 456175] 0 456175 2906 22 0 22 0 65536 608 0 rhsmcertd-worke [ 456176] 0 456176 1000 52 32 20 0 45056 44 0 bash [ 456177] 0 456177 592 33 0 33 0 36864 0 0 which [ 456178] 0 456178 446 15 0 15 0 32768 0 0 grepconf.sh [ 456179] 0 456179 1000 49 32 17 0 45056 44 0 bash [ 456180] 0 456180 2196 0 0 0 0 45056 269 -1000 sshd | Link to test (2026-10-22 09:05) |