Editing crashreport #75224

ReasonCrashing FunctionWhere to cut BacktraceReports Count
Kernel panic - not syncing: Out of memoryvpanic
panic
out_of_memory
__alloc_pages_slowpath
__alloc_pages_noprof
alloc_pages_mpol_noprof
allocate_slab
___slab_alloc
kmem_cache_alloc_noprof
spl_kmem_cache_alloc
dbuf_dirty
dnode_free_range
dmu_free_long_range_impl
dmu_free_long_range
osd_unlinked_object_free
osd_unlinked_list_emptify
osd_trans_stop
ofd_destroy
ofd_destroy_by_fid
ofd_destroy_hdl
tgt_handle_request0
tgt_request_handle
ptlrpc_server_handle_request
ptlrpc_main
kthread
ret_from_fork
3

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashLink and date
sanityn test 70a: cd directory
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 1 UID: 0 PID: 17855 Comm: ll_ost00_000 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.47.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
vpanic+0xc2/0x290
panic+0x6c/0x6c
out_of_memory.cold+0x61/0x78
__alloc_pages_slowpath.constprop.0+0x59b/0xb80
__alloc_pages_noprof+0x31d/0x330
alloc_pages_mpol_noprof+0x8d/0x1d0
allocate_slab+0x275/0x310
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_alloc+0x59/0x160 [spl]
___slab_alloc+0x65b/0xb60
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_free+0x101/0x1b0 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
kmem_cache_alloc_noprof+0xe3/0x3d0
? srso_alias_return_thunk+0x5/0xfbef5
? txg_list_add+0x7b/0x90 [zfs]
spl_kmem_cache_alloc+0x59/0x160 [spl]
dbuf_dirty+0xf6/0x970 [zfs]
? srso_alias_return_thunk+0x5/0xfbef5
dnode_free_range+0x507/0x740 [zfs]
dmu_free_long_range_impl+0x32c/0x480 [zfs]
dmu_free_long_range+0x7a/0xc0 [zfs]
osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs]
osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs]
osd_trans_stop+0x3d9/0x530 [osd_zfs]
ofd_destroy+0x38a/0x720 [ofd]
ofd_destroy_by_fid+0x309/0x470 [ofd]
ofd_destroy_hdl+0x137/0x980 [ofd]
tgt_handle_request0.isra.0+0x124/0x810 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
tgt_request_handle+0x232/0xd50 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x293/0xda0 [ptlrpc]
? lprocfs_counter_add+0x12a/0x190 [obdclass]
ptlrpc_main+0xaeb/0xed0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Crash core archive creation started: kernel-crash.onyx-155vm259.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-155vm259.vmcore.tar.gz
obd_memory max: 175790382, obd_memory current: 147443935
obd_memory max: 175790382, obd_memory current: 147443935
obd_memory max: 175790382, obd_memory current: 147452511
obd_memory max: 175790382, obd_memory current: 147452511
obd_memory max: 175790382, obd_memory current: 147456751
obd_memory max: 175790382, obd_memory current: 147444079
obd_memory max: 175790382, obd_memory current: 147443935
obd_memory max: 175790382, obd_memory current: 147443935
obd_memory max: 175790382, obd_memory current: 147443935
obd_memory max: 175790382, obd_memory current: 147444751
obd_memory max: 175790382, obd_memory current: 147444751
obd_memory max: 175790382, obd_memory current: 147444751
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147446807
obd_memory max: 175790382, obd_memory current: 147447623
obd_memory max: 175790382, obd_memory current: 147447623
obd_memory max: 175790382, obd_memory current: 147447623
obd_memory max: 175790382, obd_memory current: 147448263
obd_memory max: 175790382, obd_memory current: 147448263
obd_memory max: 175790382, obd_memory current: 147448263
obd_memory max: 175790382, obd_memory current: 147448263
obd_memory max: 175790382, obd_memory current: 147448263
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147454135
obd_memory max: 175790382, obd_memory current: 147456999
obd_memory max: 175790382, obd_memory current: 147456999
obd_memory max: 175790382, obd_memory current: 147456999
obd_memory max: 175790382, obd_memory current: 147456999
ll_ost00_000 invoked oom-killer: gfp_mask=0x42cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=0, oom_score_adj=0
CPU: 1 UID: 0 PID: 17855 Comm: ll_ost00_000 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.47.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
dump_header+0x43/0x1ad
out_of_memory.cold+0x3e/0x78
__alloc_pages_slowpath.constprop.0+0x59b/0xb80
__alloc_pages_noprof+0x31d/0x330
alloc_pages_mpol_noprof+0x8d/0x1d0
allocate_slab+0x275/0x310
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_alloc+0x59/0x160 [spl]
___slab_alloc+0x65b/0xb60
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_free+0x101/0x1b0 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
kmem_cache_alloc_noprof+0xe3/0x3d0
? srso_alias_return_thunk+0x5/0xfbef5
? txg_list_add+0x7b/0x90 [zfs]
spl_kmem_cache_alloc+0x59/0x160 [spl]
dbuf_dirty+0xf6/0x970 [zfs]
? srso_alias_return_thunk+0x5/0xfbef5
dnode_free_range+0x507/0x740 [zfs]
dmu_free_long_range_impl+0x32c/0x480 [zfs]
dmu_free_long_range+0x7a/0xc0 [zfs]
osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs]
osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs]
osd_trans_stop+0x3d9/0x530 [osd_zfs]
ofd_destroy+0x38a/0x720 [ofd]
ofd_destroy_by_fid+0x309/0x470 [ofd]
ofd_destroy_hdl+0x137/0x980 [ofd]
tgt_handle_request0.isra.0+0x124/0x810 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
tgt_request_handle+0x232/0xd50 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x293/0xda0 [ptlrpc]
? lprocfs_counter_add+0x12a/0x190 [obdclass]
ptlrpc_main+0xaeb/0xed0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Mem-Info:
active_anon:0 inactive_anon:0 isolated_anon:0
Node 0 active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:76kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:10000kB pagetables:2264kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB
Node 0 DMA free:14544kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1647 3572 3572 3572
Node 0 DMA32 free:31544kB boost:0kB min:30688kB low:38360kB high:46032kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:92kB unevictable:0kB writepending:0kB present:2080624kB managed:1687408kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1924 1924 1924
Node 0 Normal free:18172kB boost:0kB min:36604kB low:45752kB high:54900kB reserved_highatomic:4096KB active_anon:0kB inactive_anon:0kB active_file:32kB inactive_file:76kB unevictable:0kB writepending:0kB present:2097152kB managed:1970496kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 2*4kB (UE) 1*8kB (U) 2*16kB (UE) 1*32kB (E) 2*64kB (UE) 2*128kB (UE) 1*256kB (E) 1*512kB (E) 1*1024kB (E) 2*2048kB (ME) 2*4096kB (M) = 14544kB
Node 0 DMA32: 462*4kB (UME) 422*8kB (ME) 450*16kB (UME) 308*32kB (ME) 95*64kB (UME) 9*128kB (UME) 0*256kB 0*512kB 0*1024kB 1*2048kB (U) 0*4096kB = 31560kB
Node 0 Normal: 2723*4kB (UME) 578*8kB (UME) 40*16kB (UME) 23*32kB (UM) 8*64kB (UM) 3*128kB (UM) 0*256kB 1*512kB (U) 0*1024kB 0*2048kB 0*4096kB = 18300kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
42 total pagecache pages
0 pages in swap cache
Free swap = 3756476kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
130126 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 80KB 362KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 147KB 609KB
tgt_fmd_cache 11KB 27KB
tgt_session_kmem 42KB 42KB
tgt_thread_kmem 226KB 226KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 199KB 484KB
ldlm_locks 427KB 976KB
ldlm_resources 201KB 484KB
lnet_msg 63KB 63KB
lnet_rspt 360KB 360KB
lnet_small_MDs 39KB 39KB
lnet_MEs 51KB 51KB
ptlrpc_cache 802KB 802KB
ll_obd_dev_cache 318KB 318KB
bio-168 31KB 31KB
bio-248 84KB 84KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 11KB 11KB
dbuf_dirty_record_t 8793KB 8793KB
dmu_buf_impl_t 9869KB 10048KB
arc_buf_t 744KB 748KB
zio_buf_comb_16384 5024KB 5472KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 4896KB 5440KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 256KB 352KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 23KB 27KB
zio_cache 165KB 472KB
zfs_btree_leaf_cache 5432KB 6400KB
nfs_commit_data 15KB 15KB
nfs_write_data 47KB 47KB
nfs_read_data 31KB 31KB
nfs_page 8KB 8KB
netfs_subrequest 19KB 19KB
netfs_request 67KB 67KB
AF_VSOCK 15KB 15KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-128 7KB 7KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
fib6_node 8KB 8KB
ip6_dst_cache 8KB 8KB
PINGv6 30KB 30KB
RAWv6 46KB 46KB
UDPv6 31KB 31KB
TCPv6 61KB 61KB
bio-256 7KB 7KB
mqueue_inode_cache 15KB 15KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
UNIX-STREAM 154KB 207KB
UNIX 79KB 79KB
tcp_bind2_bucket 8KB 8KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 7KB 7KB
ip_fib_alias 7KB 7KB
rtable 23KB 23KB
PING 32KB 32KB
RAW 16KB 16KB
UDP 63KB 63KB
tw_sock_TCP 8KB 8KB
request_sock_TCP 7KB 7KB
TCP 123KB 123KB
hugetlbfs_inode_cache 7KB 7KB
bio-272 11KB 11KB
ep_head 8KB 8KB
eventpoll_pwq 20KB 20KB
eventpoll_epi 60KB 60KB
inotify_inode_mark 15KB 15KB
sgpool-128 128KB 128KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 8KB 8KB
bio_integrity_data 4KB 4KB
request_queue 30KB 30KB
blkdev_ioc 7KB 7KB
bio-192 60KB 64KB
biovec-max 280KB 320KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
memdup_user-1k 16KB 16KB
memdup_user-256 8KB 8KB
memdup_user-128 8KB 8KB
memdup_user-64 8KB 8KB
memdup_user-32 8KB 8KB
memdup_user-16 8KB 8KB
memdup_user-8 8KB 8KB
memdup_user-192 7KB 7KB
memdup_user-96 7KB 7KB
user_namespace 15KB 15KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_small_head 110KB 126KB
skbuff_fclone_cache 64KB 64KB
skbuff_head_cache 28KB 40KB
configfs_dir_cache 3KB 3KB
file_lock_cache 7KB 7KB
file_lock_ctx 15KB 15KB
fsnotify_mark_connector 7KB 7KB
posix_timers_cache 55KB 55KB
taskstats 15KB 15KB
proc_dir_entry 216KB 216KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1078KB 1087KB
kernfs_iattrs_cache 15KB 15KB
kernfs_node_cache 4356KB 4390KB
mnt_cache 231KB 252KB
filp 223KB 244KB
names_cache 160KB 160KB
net_namespace 62KB 62KB
lsm_inode_cache 414KB 968KB
lsm_file_cache 25KB 32KB
key_jar 24KB 24KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 169KB 187KB
vm_area_struct 743KB 743KB
fs_cache 20KB 20KB
files_cache 59KB 110KB
signal_cache 696KB 740KB
sighand_cache 1274KB 1361KB
task_struct 5340KB 5492KB
cred 149KB 157KB
anon_vma_chain 145KB 160KB
anon_vma 210KB 211KB
pid 126KB 133KB
Acpi-Operand 122KB 122KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 23KB 23KB
Acpi-Namespace 19KB 19KB
numa_policy 3KB 3KB
perf_event 15KB 15KB
trace_event_file 220KB 220KB
ftrace_event_field 339KB 339KB
pool_workqueue 112KB 112KB
maple_node 201KB 304KB
task_group 90KB 90KB
mm_struct 187KB 187KB
vmap_area 456KB 653KB
kmalloc_buckets 3KB 3KB
kmalloc-cg-4k 224KB 288KB
kmalloc-cg-2k 672KB 832KB
kmalloc-cg-1k 159KB 192KB
kmalloc-cg-512 64KB 64KB
kmalloc-cg-256 20KB 20KB
kmalloc-cg-128 23KB 36KB
kmalloc-cg-64 14KB 36KB
kmalloc-cg-32 21KB 28KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 16KB 16KB
kmalloc-cg-192 44KB 59KB
kmalloc-cg-96 30KB 51KB
kmalloc-8k 6952KB 7392KB
kmalloc-4k 14240KB 14240KB
kmalloc-2k 2690KB 2784KB
kmalloc-1k 9940KB 10448KB
kmalloc-512 2993KB 4056KB
kmalloc-256 631KB 740KB
kmalloc-128 413KB 820KB
kmalloc-64 1293KB 1488KB
kmalloc-32 448KB 648KB
kmalloc-16 162KB 168KB
kmalloc-8 51KB 52KB
kmalloc-192 2638KB 2744KB
kmalloc-96 1178KB 1720KB
kmem_cache_node 28KB 28KB
kmem_cache 116KB 116KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 657] 0 657 10418 23 0 18 5 86016 320 -250 systemd-journal
[ 694] 0 694 3979 25 0 25 0 69632 224 0 systemd-userdbd
[ 705] 0 705 8512 68 32 36 0 90112 480 -1000 systemd-udevd
[ 769] 32 769 3163 56 32 24 0 65536 224 0 rpcbind
[ 770] 0 770 5105 42 4 38 0 61440 224 -1000 auditd
[ 802] 81 802 2203 55 32 23 0 53248 128 -900 dbus-broker-lau
[ 804] 81 804 1224 40 0 40 0 49152 96 -900 dbus-broker
[ 805] 0 805 82716 33 0 33 0 143360 704 0 NetworkManager
[ 806] 994 806 21172 68 32 36 0 69632 128 0 chronyd
[ 808] 0 808 20106 49 0 49 0 53248 64 0 irqbalance
[ 810] 0 810 4196 51 32 19 0 73728 256 0 systemd-logind
[ 836] 0 836 30676 41 32 9 0 94208 512 0 gssproxy
[ 839] 0 839 2196 55 32 23 0 61440 256 -1000 sshd
[ 848] 0 848 20281 32 0 32 0 53248 32 0 rhsmcertd
[ 890] 0 890 42810 65 32 33 0 94208 192 0 rsyslogd
[ 899] 29 899 2654 20 10 10 0 53248 192 0 rpc.statd
[ 902] 996 902 19223 44 40 4 0 69632 320 0 munged
[ 955] 0 955 84319 66 32 34 0 114688 480 0 automount
[ 967] 0 967 1248 74 32 42 0 49152 32 0 agetty
[ 970] 0 970 1890 29 0 29 0 53248 32 0 agetty
[ 998] 0 998 2077 32 0 32 0 53248 192 0 crond
[ 1321] 0 1321 3795 8 0 8 0 65536 480 0 sshd-session
[ 1326] 0 1326 5173 38 0 38 0 81920 512 100 systemd
[ 1328] 0 1328 3415 41 29 12 0 57344 288 100 (sd-pam)
[ 1337] 0 1337 3885 43 0 43 0 69632 544 0 sshd-session
[ 270141] 0 270141 3795 49 32 17 0 61440 480 0 sshd-session
[ 270144] 0 270144 3887 23 3 20 0 61440 576 0 sshd-session
[ 293938] 0 293938 4092 72 32 40 0 73728 224 0 systemd-userwor
[ 293939] 0 293939 4092 72 32 40 0 73728 224 0 systemd-userwor
[ 295087] 0 295087 4092 74 32 42 0 69632 224 0 systemd-userwor
[ 298316] 0 298316 3887 41 13 28 0 61440 545 0 sshd-session
Link to test (2026-09-03 06:49)
sanityn test 70a: cd directory
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 0 UID: 0 PID: 172345 Comm: ll_ost00_079 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.47.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
vpanic+0xc2/0x290
panic+0x6c/0x6c
out_of_memory.cold+0x61/0x78
__alloc_pages_slowpath.constprop.0+0x59b/0xb80
__alloc_pages_noprof+0x31d/0x330
alloc_pages_mpol_noprof+0x8d/0x1d0
allocate_slab+0x275/0x310
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_alloc+0x59/0x160 [spl]
___slab_alloc+0x65b/0xb60
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_free+0x101/0x1b0 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
kmem_cache_alloc_noprof+0xe3/0x3d0
? srso_alias_return_thunk+0x5/0xfbef5
? txg_list_add+0x7b/0x90 [zfs]
spl_kmem_cache_alloc+0x59/0x160 [spl]
dbuf_dirty+0xf6/0x970 [zfs]
? srso_alias_return_thunk+0x5/0xfbef5
dnode_free_range+0x507/0x740 [zfs]
dmu_free_long_range_impl+0x32c/0x480 [zfs]
dmu_free_long_range+0x7a/0xc0 [zfs]
osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs]
osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs]
osd_trans_stop+0x3d9/0x530 [osd_zfs]
ofd_destroy+0x38a/0x720 [ofd]
ofd_destroy_by_fid+0x309/0x470 [ofd]
ofd_destroy_hdl+0x137/0x980 [ofd]
tgt_handle_request0.isra.0+0x124/0x810 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
tgt_request_handle+0x232/0xd50 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x293/0xda0 [ptlrpc]
? lprocfs_counter_add+0x12a/0x190 [obdclass]
ptlrpc_main+0xaeb/0xed0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Crash core archive creation started: kernel-crash.onyx-158vm220.vmcore.tar.gz
Crash core archive created: kernel-crash.onyx-158vm220.vmcore.tar.gz
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
obd_memory max: 176568375, obd_memory current: 145222595
ll_ost00_079 invoked oom-killer: gfp_mask=0x42cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=0, oom_score_adj=0
CPU: 0 UID: 0 PID: 172345 Comm: ll_ost00_079 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.47.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
dump_header+0x43/0x1ad
out_of_memory.cold+0x3e/0x78
__alloc_pages_slowpath.constprop.0+0x59b/0xb80
__alloc_pages_noprof+0x31d/0x330
alloc_pages_mpol_noprof+0x8d/0x1d0
allocate_slab+0x275/0x310
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_alloc+0x59/0x160 [spl]
___slab_alloc+0x65b/0xb60
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? spl_kmem_cache_free+0x101/0x1b0 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
kmem_cache_alloc_noprof+0xe3/0x3d0
? srso_alias_return_thunk+0x5/0xfbef5
? txg_list_add+0x7b/0x90 [zfs]
spl_kmem_cache_alloc+0x59/0x160 [spl]
dbuf_dirty+0xf6/0x970 [zfs]
? srso_alias_return_thunk+0x5/0xfbef5
dnode_free_range+0x507/0x740 [zfs]
dmu_free_long_range_impl+0x32c/0x480 [zfs]
dmu_free_long_range+0x7a/0xc0 [zfs]
osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs]
osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs]
osd_trans_stop+0x3d9/0x530 [osd_zfs]
ofd_destroy+0x38a/0x720 [ofd]
ofd_destroy_by_fid+0x309/0x470 [ofd]
ofd_destroy_hdl+0x137/0x980 [ofd]
tgt_handle_request0.isra.0+0x124/0x810 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
tgt_request_handle+0x232/0xd50 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x293/0xda0 [ptlrpc]
? lprocfs_counter_add+0x12a/0x190 [obdclass]
ptlrpc_main+0xaeb/0xed0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xfa/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x31/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Mem-Info:
active_anon:4 inactive_anon:8 isolated_anon:0
Node 0 active_anon:16kB inactive_anon:32kB active_file:64kB inactive_file:44kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:60kB dirty:24kB writeback:0kB shmem:20kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:9968kB pagetables:2304kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB
Node 0 DMA free:14572kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1647 3572 3572 3572
Node 0 DMA32 free:38144kB boost:0kB min:30688kB low:38360kB high:46032kB reserved_highatomic:0KB active_anon:0kB inactive_anon:28kB active_file:0kB inactive_file:48kB unevictable:0kB writepending:0kB present:2080624kB managed:1687408kB mlocked:0kB bounce:0kB free_pcp:712kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1924 1924 1924
Node 0 Normal free:19100kB boost:0kB min:36604kB low:45752kB high:54900kB reserved_highatomic:0KB active_anon:0kB inactive_anon:88kB active_file:0kB inactive_file:44kB unevictable:0kB writepending:24kB present:2097152kB managed:1970496kB mlocked:0kB bounce:0kB free_pcp:332kB local_pcp:224kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 3*4kB (UE) 2*8kB (UE) 2*16kB (UE) 2*32kB (UE) 2*64kB (UE) 2*128kB (UE) 1*256kB (E) 1*512kB (E) 1*1024kB (E) 2*2048kB (ME) 2*4096kB (M) = 14588kB
Node 0 DMA32: 88*4kB (UME) 278*8kB (UME) 594*16kB (UME) 279*32kB (UME) 128*64kB (UME) 46*128kB (UME) 6*256kB (UM) 1*512kB (E) 1*1024kB (U) 0*2048kB 0*4096kB = 38160kB
Node 0 Normal: 781*4kB (UME) 439*8kB (ME) 320*16kB (ME) 133*32kB (ME) 41*64kB (UME) 3*128kB (E) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 19020kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
31 total pagecache pages
0 pages in swap cache
Free swap = 3756372kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
130126 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 68KB 215KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 125KB 424KB
vvp_thread_kmem 15KB 15KB
vvp_session_kmem 7KB 7KB
ll_thread_kmem 7KB 7KB
lov_session_kmem 15KB 15KB
lov_thread_kmem 7KB 7KB
osc_session_kmem 7KB 7KB
osc_thread_kmem 58KB 58KB
tgt_fmd_cache 11KB 27KB
tgt_session_kmem 62KB 62KB
tgt_thread_kmem 226KB 226KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 149KB 460KB
ldlm_locks 301KB 976KB
ldlm_resources 154KB 472KB
lnet_msg 63KB 63KB
lnet_rspt 2668KB 2668KB
lnet_small_MDs 39KB 39KB
lnet_MEs 52KB 55KB
ptlrpc_cache 689KB 787KB
cl_env_kmem 3KB 3KB
ll_obd_dev_cache 318KB 318KB
bio-168 31KB 31KB
bio-248 76KB 76KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 11KB 11KB
dbuf_dirty_record_t 8352KB 8396KB
dmu_buf_impl_t 10337KB 10380KB
arc_buf_t 740KB 744KB
zio_buf_comb_16384 4864KB 5408KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 4152KB 6144KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 192KB 288KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 227KB 231KB
zio_cache 5203KB 5310KB
zfs_btree_leaf_cache 2944KB 6176KB
nfs_commit_data 31KB 31KB
nfs_write_data 95KB 175KB
nfs_read_data 31KB 31KB
nfs_page 44KB 44KB
netfs_subrequest 19KB 19KB
netfs_request 67KB 67KB
AF_VSOCK 30KB 30KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-128 7KB 7KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
fib6_node 8KB 8KB
ip6_dst_cache 8KB 8KB
PINGv6 15KB 15KB
RAWv6 30KB 30KB
UDPv6 31KB 31KB
TCPv6 61KB 61KB
bio-256 7KB 7KB
mqueue_inode_cache 15KB 15KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
UNIX-STREAM 191KB 191KB
UNIX 79KB 79KB
tcp_bind2_bucket 8KB 8KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 7KB 7KB
ip_fib_alias 7KB 7KB
rtable 23KB 23KB
PING 16KB 16KB
RAW 16KB 16KB
UDP 63KB 63KB
tw_sock_TCP 8KB 8KB
request_sock_TCP 7KB 7KB
TCP 123KB 123KB
hugetlbfs_inode_cache 15KB 15KB
bio-272 11KB 11KB
ep_head 8KB 8KB
eventpoll_pwq 16KB 16KB
eventpoll_epi 44KB 52KB
inotify_inode_mark 15KB 15KB
sgpool-128 160KB 160KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 8KB 8KB
bio_integrity_data 4KB 4KB
request_queue 30KB 30KB
blkdev_ioc 7KB 7KB
bio-192 40KB 44KB
biovec-max 320KB 320KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
memdup_user-1k 16KB 16KB
memdup_user-256 8KB 8KB
memdup_user-128 8KB 8KB
memdup_user-64 8KB 8KB
memdup_user-32 8KB 8KB
memdup_user-16 8KB 8KB
memdup_user-8 8KB 8KB
memdup_user-192 7KB 7KB
memdup_user-96 7KB 7KB
user_namespace 15KB 15KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_small_head 126KB 126KB
skbuff_fclone_cache 64KB 64KB
skbuff_head_cache 48KB 48KB
configfs_dir_cache 3KB 3KB
file_lock_cache 7KB 7KB
file_lock_ctx 15KB 15KB
fsnotify_mark_connector 7KB 7KB
posix_timers_cache 55KB 55KB
taskstats 15KB 15KB
proc_dir_entry 220KB 220KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1045KB 1087KB
kernfs_iattrs_cache 19KB 19KB
kernfs_node_cache 4494KB 4510KB
mnt_cache 225KB 244KB
filp 232KB 259KB
names_cache 192KB 192KB
net_namespace 31KB 31KB
lsm_inode_cache 402KB 948KB
lsm_file_cache 23KB 36KB
key_jar 28KB 28KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 168KB 195KB
vm_area_struct 714KB 735KB
fs_cache 16KB 16KB
files_cache 110KB 110KB
signal_cache 698KB 740KB
sighand_cache 1307KB 1361KB
task_struct 5324KB 5518KB
cred 138KB 153KB
anon_vma_chain 132KB 160KB
anon_vma 196KB 199KB
pid 130KB 137KB
Acpi-Operand 122KB 122KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 23KB 23KB
Acpi-Namespace 19KB 19KB
numa_policy 3KB 3KB
perf_event 15KB 15KB
trace_event_file 220KB 220KB
ftrace_event_field 339KB 339KB
pool_workqueue 112KB 112KB
maple_node 206KB 316KB
task_group 82KB 82KB
mm_struct 146KB 218KB
vmap_area 453KB 633KB
kmalloc_buckets 3KB 3KB
kmalloc-cg-8k 32KB 32KB
kmalloc-cg-4k 288KB 288KB
kmalloc-cg-2k 668KB 800KB
kmalloc-cg-1k 150KB 192KB
kmalloc-cg-512 64KB 64KB
kmalloc-cg-256 24KB 24KB
kmalloc-cg-128 31KB 44KB
kmalloc-cg-64 31KB 48KB
kmalloc-cg-32 22KB 28KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 12KB 12KB
kmalloc-cg-192 70KB 70KB
kmalloc-cg-96 51KB 66KB
kmalloc-8k 6864KB 7552KB
kmalloc-4k 14104KB 14112KB
kmalloc-2k 2724KB 2816KB
kmalloc-1k 8265KB 10464KB
kmalloc-512 2555KB 4464KB
kmalloc-256 1183KB 1232KB
kmalloc-128 676KB 872KB
kmalloc-64 1242KB 1656KB
kmalloc-32 426KB 716KB
kmalloc-16 167KB 172KB
kmalloc-8 51KB 52KB
kmalloc-192 2635KB 2748KB
kmalloc-96 1103KB 1827KB
kmem_cache_node 28KB 28KB
kmem_cache 120KB 120KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 655] 0 655 10418 36 0 36 0 81920 320 -250 systemd-journal
[ 692] 0 692 3979 72 32 40 0 77824 224 0 systemd-userdbd
[ 702] 0 702 8545 63 32 31 0 86016 512 -1000 systemd-udevd
[ 766] 32 766 3163 9 0 9 0 69632 224 0 rpcbind
[ 767] 0 767 5105 84 28 56 0 61440 192 -1000 auditd
[ 799] 81 799 2203 58 32 26 0 57344 128 -900 dbus-broker-lau
[ 802] 81 802 1224 75 32 43 0 45056 96 -900 dbus-broker
[ 803] 0 803 82717 87 32 55 0 147456 672 0 NetworkManager
[ 804] 994 804 21172 61 32 29 0 65536 160 0 chronyd
[ 806] 0 806 20106 45 0 45 0 61440 64 0 irqbalance
[ 809] 0 809 4192 71 32 39 0 81920 288 0 systemd-logind
[ 831] 0 831 30676 64 32 32 0 94208 512 0 gssproxy
[ 836] 0 836 2196 69 32 37 0 61440 256 -1000 sshd
[ 843] 0 843 20281 51 0 51 0 57344 32 0 rhsmcertd
[ 889] 0 889 42810 27 0 27 0 94208 224 0 rsyslogd
[ 898] 996 898 19223 57 40 17 0 61440 288 0 munged
[ 899] 29 899 2654 32 10 22 0 57344 192 0 rpc.statd
[ 954] 0 954 84320 78 32 46 0 122880 480 0 automount
[ 967] 0 967 1248 31 0 31 0 49152 32 0 agetty
[ 968] 0 968 1890 17 0 17 0 53248 64 0 agetty
[ 985] 0 985 2076 23 0 23 0 61440 192 0 crond
[ 1319] 0 1319 3795 27 0 27 0 69632 480 0 sshd-session
[ 1324] 0 1324 5174 70 32 38 0 81920 480 100 systemd
[ 1326] 0 1326 3415 59 28 31 0 65536 288 100 (sd-pam)
[ 1335] 0 1335 3885 45 15 30 0 69632 512 0 sshd-session
[ 270431] 0 270431 3795 52 32 20 0 73728 448 0 sshd-session
[ 270434] 0 270434 3922 24 18 6 0 73728 544 0 sshd-session
[ 296915] 0 296915 4092 69 32 37 0 73728 224 0 systemd-userwor
[ 297680] 0 297680 4092 42 32 10 0 73728 224 0 systemd-userwor
[ 297681] 0 297681 4092 71 32 39 0 69632 224 0 systemd-userwor
[ 298447] 0 298447 1002 41 32 9 0 45056 64 0 bash
[ 298456] 0 298456 1002 37 33 4 0 45056 64 0 bash
Link to test (2026-09-03 06:24)
sanityn test 60: Verify data_version behaviour
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 0 UID: 0 PID: 2089394 Comm: ll_ost00_002 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.40.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
vpanic+0xc2/0x290
panic+0x6c/0x6c
out_of_memory.cold+0x61/0x78
__alloc_pages_slowpath.constprop.0+0x59b/0xb80
__alloc_pages_noprof+0x31d/0x330
alloc_pages_mpol_noprof+0x8d/0x1d0
allocate_slab+0x275/0x310
? spl_kmem_cache_alloc+0x59/0x160 [spl]
___slab_alloc+0x65b/0xb60
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_free+0x101/0x1b0 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
kmem_cache_alloc_noprof+0xe3/0x3d0
? _raw_spin_unlock+0xe/0x30
? txg_list_add+0x7b/0x90 [zfs]
spl_kmem_cache_alloc+0x59/0x160 [spl]
dbuf_dirty+0xf6/0x970 [zfs]
dnode_free_range+0x507/0x740 [zfs]
dmu_free_long_range_impl+0x32c/0x480 [zfs]
dmu_free_long_range+0x7a/0xc0 [zfs]
osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs]
osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs]
osd_trans_stop+0x3e9/0x540 [osd_zfs]
ofd_destroy+0x38a/0x720 [ofd]
ofd_destroy_by_fid+0x309/0x470 [ofd]
ofd_destroy_hdl+0x137/0x980 [ofd]
tgt_handle_request0.isra.0+0x127/0x810 [ptlrpc]
tgt_request_handle+0x232/0xd50 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x296/0xda0 [ptlrpc]
? lprocfs_counter_add+0x12a/0x190 [obdclass]
ptlrpc_main+0xaeb/0xed0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xfd/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146779014
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
obd_memory max: 151128880, obd_memory current: 146781926
ll_ost00_002 invoked oom-killer: gfp_mask=0x42cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=0, oom_score_adj=0
CPU: 0 UID: 0 PID: 2089394 Comm: ll_ost00_002 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.40.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary)
Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x4e/0x70
dump_header+0x43/0x1ad
out_of_memory.cold+0x3e/0x78
__alloc_pages_slowpath.constprop.0+0x59b/0xb80
__alloc_pages_noprof+0x31d/0x330
alloc_pages_mpol_noprof+0x8d/0x1d0
allocate_slab+0x275/0x310
? spl_kmem_cache_alloc+0x59/0x160 [spl]
___slab_alloc+0x65b/0xb60
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
? spl_kmem_cache_free+0x101/0x1b0 [spl]
? spl_kmem_cache_alloc+0x59/0x160 [spl]
kmem_cache_alloc_noprof+0xe3/0x3d0
? _raw_spin_unlock+0xe/0x30
? txg_list_add+0x7b/0x90 [zfs]
spl_kmem_cache_alloc+0x59/0x160 [spl]
dbuf_dirty+0xf6/0x970 [zfs]
dnode_free_range+0x507/0x740 [zfs]
dmu_free_long_range_impl+0x32c/0x480 [zfs]
dmu_free_long_range+0x7a/0xc0 [zfs]
osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs]
osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs]
osd_trans_stop+0x3e9/0x540 [osd_zfs]
ofd_destroy+0x38a/0x720 [ofd]
ofd_destroy_by_fid+0x309/0x470 [ofd]
ofd_destroy_hdl+0x137/0x980 [ofd]
tgt_handle_request0.isra.0+0x127/0x810 [ptlrpc]
tgt_request_handle+0x232/0xd50 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x296/0xda0 [ptlrpc]
? lprocfs_counter_add+0x12a/0x190 [obdclass]
ptlrpc_main+0xaeb/0xed0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xfd/0x240
? __pfx_kthread+0x10/0x10
ret_from_fork+0x34/0x50
? __pfx_kthread+0x10/0x10
ret_from_fork_asm+0x1a/0x30
</TASK>
Mem-Info:
active_anon:6 inactive_anon:0 isolated_anon:0
Node 0 active_anon:24kB inactive_anon:0kB active_file:0kB inactive_file:144kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:16kB dirty:0kB writeback:20kB shmem:8kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:8912kB pagetables:2652kB sec_pagetables:0kB all_unreclaimable? yes Balloon:0kB
Node 0 DMA free:14572kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1647 3574 3574 3574
Node 0 DMA32 free:37076kB boost:0kB min:30672kB low:38340kB high:46008kB reserved_highatomic:0KB active_anon:68kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:8kB present:2080624kB managed:1687408kB mlocked:0kB bounce:0kB free_pcp:16kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1926 1926 1926
Node 0 Normal free:31136kB boost:0kB min:36620kB low:45772kB high:54924kB reserved_highatomic:0KB active_anon:16kB inactive_anon:0kB active_file:24kB inactive_file:60kB unevictable:0kB writepending:12kB present:2097152kB managed:1972520kB mlocked:0kB bounce:0kB free_pcp:136kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 3*4kB (UE) 4*8kB (UE) 2*16kB (UE) 1*32kB (E) 2*64kB (UE) 2*128kB (UE) 1*256kB (E) 1*512kB (E) 1*1024kB (E) 2*2048kB (ME) 2*4096kB (M) = 14572kB
Node 0 DMA32: 529*4kB (UME) 482*8kB (UME) 437*16kB (UME) 278*32kB (ME) 150*64kB (UME) 42*128kB (ME) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37092kB
Node 0 Normal: 649*4kB (UME) 944*8kB (UME) 512*16kB (UME) 214*32kB (UME) 94*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 31204kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
30 total pagecache pages
10 pages in swap cache
Free swap = 3747220kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
129620 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 87KB 227KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 123KB 354KB
vvp_thread_kmem 15KB 15KB
vvp_session_kmem 7KB 7KB
ll_thread_kmem 7KB 7KB
lov_session_kmem 15KB 15KB
lov_thread_kmem 7KB 7KB
osc_session_kmem 7KB 7KB
osc_thread_kmem 58KB 58KB
tgt_fmd_cache 7KB 7KB
tgt_session_kmem 54KB 54KB
tgt_thread_kmem 196KB 196KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 333KB 352KB
ldlm_locks 638KB 856KB
ldlm_resources 333KB 352KB
lnet_msg 55KB 55KB
lnet_rspt 1280KB 1280KB
lnet_small_MDs 43KB 59KB
lnet_MEs 47KB 51KB
ptlrpc_cache 648KB 648KB
cl_env_kmem 3KB 3KB
ll_obd_dev_cache 287KB 319KB
bio-168 35KB 35KB
bio-248 80KB 80KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 7KB 7KB
zil_zcw_cache 7KB 7KB
zil_lwb_cache 15KB 15KB
dbuf_dirty_record_t 8254KB 8266KB
dmu_buf_impl_t 9770KB 9780KB
arc_buf_t 731KB 732KB
zio_buf_comb_16384 4640KB 5248KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 920KB 2240KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 216KB 352KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 20KB 35KB
zio_cache 228KB 409KB
zfs_btree_leaf_cache 1788KB 3808KB
nfs_commit_data 15KB 15KB
nfs_write_data 47KB 47KB
nfs_read_data 31KB 31KB
nfs_page 8KB 8KB
netfs_subrequest 19KB 19KB
netfs_request 67KB 67KB
AF_VSOCK 30KB 30KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-128 7KB 7KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
fib6_node 8KB 8KB
ip6_dst_cache 8KB 8KB
PINGv6 30KB 30KB
RAWv6 30KB 30KB
UDPv6 31KB 31KB
TCPv6 61KB 61KB
bio-256 7KB 7KB
mqueue_inode_cache 15KB 15KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
UNIX-STREAM 159KB 191KB
UNIX 79KB 79KB
tcp_bind2_bucket 8KB 8KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 7KB 7KB
ip_fib_alias 7KB 7KB
rtable 31KB 31KB
PING 32KB 32KB
RAW 16KB 16KB
UDP 78KB 78KB
tw_sock_TCP 8KB 8KB
request_sock_TCP 7KB 7KB
TCP 173KB 277KB
hugetlbfs_inode_cache 15KB 15KB
bio-272 11KB 11KB
ep_head 8KB 8KB
eventpoll_pwq 32KB 32KB
eventpoll_epi 31KB 52KB
inotify_inode_mark 15KB 15KB
sgpool-128 128KB 128KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 8KB 8KB
bio_integrity_data 4KB 4KB
request_queue 30KB 30KB
blkdev_ioc 7KB 7KB
bio-192 52KB 52KB
biovec-max 312KB 352KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
memdup_user-1k 16KB 16KB
memdup_user-256 8KB 8KB
memdup_user-128 8KB 8KB
memdup_user-64 8KB 8KB
memdup_user-32 8KB 8KB
memdup_user-16 8KB 8KB
memdup_user-8 8KB 8KB
memdup_user-192 7KB 7KB
memdup_user-96 7KB 7KB
user_namespace 15KB 15KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_small_head 47KB 126KB
skbuff_fclone_cache 64KB 64KB
skbuff_head_cache 48KB 48KB
configfs_dir_cache 7KB 7KB
file_lock_cache 7KB 7KB
file_lock_ctx 15KB 15KB
fsnotify_mark_connector 7KB 7KB
posix_timers_cache 55KB 55KB
taskstats 15KB 15KB
proc_dir_entry 196KB 196KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1095KB 1225KB
kernfs_iattrs_cache 19KB 19KB
kernfs_node_cache 4484KB 4518KB
mnt_cache 204KB 204KB
filp 214KB 248KB
names_cache 160KB 224KB
net_namespace 62KB 62KB
lsm_inode_cache 401KB 988KB
lsm_file_cache 25KB 32KB
key_jar 28KB 28KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 168KB 191KB
vm_area_struct 708KB 731KB
fs_cache 20KB 20KB
files_cache 115KB 126KB
signal_cache 614KB 630KB
sighand_cache 1126KB 1144KB
task_struct 3869KB 3869KB
cred 129KB 133KB
anon_vma_chain 135KB 152KB
anon_vma 200KB 203KB
pid 118KB 118KB
Acpi-Operand 122KB 122KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 23KB 23KB
Acpi-Namespace 19KB 19KB
numa_policy 3KB 3KB
perf_event 15KB 15KB
trace_event_file 220KB 220KB
ftrace_event_field 339KB 339KB
pool_workqueue 144KB 144KB
maple_node 215KB 340KB
task_group 90KB 90KB
mm_struct 218KB 218KB
vmap_area 449KB 673KB
kmalloc_buckets 3KB 3KB
kmalloc-cg-4k 252KB 288KB
kmalloc-cg-2k 682KB 832KB
kmalloc-cg-1k 159KB 208KB
kmalloc-cg-512 64KB 64KB
kmalloc-cg-256 20KB 20KB
kmalloc-cg-128 32KB 48KB
kmalloc-cg-64 18KB 36KB
kmalloc-cg-32 13KB 28KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 13KB 16KB
kmalloc-cg-192 51KB 51KB
kmalloc-cg-96 44KB 51KB
kmalloc-8k 6128KB 6912KB
kmalloc-4k 13924KB 13924KB
kmalloc-2k 2400KB 2464KB
kmalloc-1k 7270KB 8624KB
kmalloc-512 4441KB 5984KB
kmalloc-256 671KB 900KB
kmalloc-128 506KB 784KB
kmalloc-64 1227KB 1600KB
kmalloc-32 418KB 684KB
kmalloc-16 172KB 176KB
kmalloc-8 51KB 52KB
kmalloc-192 5288KB 5414KB
kmalloc-96 1188KB 1862KB
kmem_cache_node 32KB 32KB
kmem_cache 146KB 146KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name
[ 622] 0 622 10341 66 0 47 19 90112 416 -250 systemd-journal
[ 658] 0 658 3979 81 32 49 0 73728 224 0 systemd-userdbd
[ 669] 0 669 8561 67 32 35 0 94208 544 -1000 systemd-udevd
[ 733] 32 733 3163 52 32 20 0 61440 192 0 rpcbind
[ 734] 0 734 5105 52 36 16 0 61440 192 -1000 auditd
[ 766] 81 766 2203 58 32 26 0 61440 128 -900 dbus-broker-lau
[ 768] 81 768 1225 63 32 31 0 49152 96 -900 dbus-broker
[ 769] 0 769 82718 70 32 38 0 147456 704 0 NetworkManager
[ 770] 994 770 21172 51 0 51 0 69632 160 0 chronyd
[ 772] 0 772 20106 78 32 46 0 57344 64 0 irqbalance
[ 774] 0 774 4196 81 32 49 0 73728 288 0 systemd-logind
[ 795] 0 795 30676 32 0 32 0 98304 544 0 gssproxy
[ 801] 0 801 2196 49 0 49 0 57344 256 -1000 sshd
[ 803] 0 803 20281 18 0 18 0 57344 32 0 rhsmcertd
[ 861] 0 861 65161 107 32 41 34 122880 256 0 rsyslogd
[ 868] 29 868 2654 45 10 35 0 53248 192 0 rpc.statd
[ 869] 996 869 19223 25 8 17 0 65536 320 0 munged
[ 919] 0 919 84318 81 32 49 0 118784 480 0 automount
[ 932] 0 932 1248 29 0 29 0 57344 32 0 agetty
[ 933] 0 933 1890 13 0 13 0 57344 32 0 agetty
[ 942] 0 942 2078 42 0 42 0 53248 192 0 crond
[ 1265] 0 1265 3795 80 32 48 0 69632 448 0 sshd-session
[ 1270] 0 1270 5173 71 32 39 0 81920 512 100 systemd
[ 1272] 0 1272 3415 20 3 17 0 61440 288 100 (sd-pam)
[ 1281] 0 1281 3885 63 44 19 0 73728 544 0 sshd-session
[2164138] 0 2164138 3795 61 32 29 0 69632 448 0 sshd-session
[2164141] 0 2164141 3887 36 14 22 0 69632 544 0 sshd-session
[2186020] 0 2186020 4092 49 0 49 0 77824 224 0 systemd-userwor
[2187934] 0 2187934 4092 46 0 46 0 81920 224 0 systemd-userwor
[2187935] 0 2187935 4092 53 32 21 0 73728 224 0 systemd-userwor
[2192085] 0 2192085 1002 42 0 42 0 53248 64 0 bash
[2192091] 0 2192091 980 32 0 32 0 36864 32 0 grepconf.sh
Link to test (2026-08-01 09:32)
Return to new crashes list