Editing crashreport #75076

ReasonCrashing FunctionWhere to cut BacktraceReports Count
Kernel panic - not syncing: Out of memoryvpanic
panic
out_of_memory
__alloc_pages_slowpath
__alloc_pages
alloc_pages_mpol
__read_swap_cache_async
swap_cluster_readahead
swapin_readahead
do_swap_page
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
4

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanityn test 70b: remove files after calling rm_entry
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 1 PID: 1582536 Comm: tclsh Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.26.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
vpanic+0x129/0x2f0
panic+0x58/0x60
out_of_memory+0x2ae/0x330
__alloc_pages_slowpath.constprop.0+0xa58/0xc60
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
? srso_alias_return_thunk+0x5/0xfbef5
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? pick_next_task_fair+0x1d3/0x510
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x408/0xac0
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x1fe/0x690
exc_page_fault+0x74/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0010:__get_user_8+0x16/0x30
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
obd_memory max: 152820100, obd_memory current: 148492698
tclsh invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 1 PID: 1582536 Comm: tclsh Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.26.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
dump_header+0x49/0x240
out_of_memory+0x289/0x330
__alloc_pages_slowpath.constprop.0+0xa58/0xc60
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
? srso_alias_return_thunk+0x5/0xfbef5
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? pick_next_task_fair+0x1d3/0x510
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x408/0xac0
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x1fe/0x690
exc_page_fault+0x74/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0010:__get_user_8+0x16/0x30
Code: 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 ba 00 f0 ff ff ff ff ff 00 48 39 c2 48 19 d2 48 09 d0 0f 01 cb <48> 8b 10 31 c0 0f 01 ca e9 b8 7e 7d 00 66 66 2e 0f 1f 84 00 00 00
RSP: 0018:ff764ffdc20b3e50 EFLAGS: 00050202
RAX: 00007fb301087da8 RBX: ff483dbb7db1c680 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff764ffdc20b3e80 RDI: ff483dbb7db1c680
RBP: ff764ffdc20b3ef0 R08: 0000000000000100 R09: 0000000000000012
R10: ff483dbb84c10e0c R11: ff483dba871bebb0 R12: 0000000000000002
R13: 00007fb301176f80 R14: ff764ffdc20b3f58 R15: 0000000000000000
rseq_get_rseq_cs+0x1d/0x230
rseq_ip_fixup+0x45/0x1b0
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x138/0x270
__rseq_handle_notify_resume+0x25/0xc0
exit_to_user_mode_loop+0xd8/0x130
exit_to_user_mode_prepare+0x6b/0x80
irqentry_exit_to_user_mode+0x5/0x30
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7fb301176f80
Code: Unable to access opcode bytes at 0x7fb301176f56.
RSP: 002b:00007ffe2bc619b0 EFLAGS: 00010202
RAX: 000000000000003a RBX: 0000000000000000 RCX: 0000560c6d34c670
RDX: 0000000000000000 RSI: 0000560c6d3469e0 RDI: 0000560c6d3367b0
RBP: 0000560c6d3367b0 R08: 0000560c6d2e0290 R09: 0000560c6d43e780
R10: 00007ffe2bc61888 R11: 00007fb300ffbca0 R12: 0000560c6d3d5180
R13: 0000000000000000 R14: 000000006d3469e0 R15: 0000560c6d3469e0
</TASK>
Mem-Info:
active_anon:0 inactive_anon:3 isolated_anon:0
Node 0 active_anon:0kB inactive_anon:12kB active_file:36kB inactive_file:68kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4kB dirty:0kB writeback:0kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:8848kB pagetables:1916kB sec_pagetables:0kB all_unreclaimable? no
Node 0 DMA free:14848kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:2048KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1678 3598 3598 3598
Node 0 DMA32 free:37216kB boost:0kB min:31388kB low:39232kB high:47076kB reserved_highatomic:0KB active_anon:0kB inactive_anon:12kB active_file:184kB inactive_file:0kB unevictable:0kB writepending:0kB present:2080624kB managed:1752944kB mlocked:0kB bounce:0kB free_pcp:760kB local_pcp:488kB free_cma:0kB
lowmem_reserve[]: 0 0 1920 1920 1920
Node 0 Normal free:17780kB boost:0kB min:35912kB low:44888kB high:53864kB reserved_highatomic:2048KB active_anon:0kB inactive_anon:0kB active_file:244kB inactive_file:0kB unevictable:0kB writepending:0kB present:2097152kB managed:1972900kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 1*512kB (U) 0*1024kB 1*2048kB (M) 3*4096kB (UM) = 14848kB
Node 0 DMA32: 246*4kB (UME) 208*8kB (ME) 198*16kB (UME) 127*32kB (UME) 83*64kB (ME) 18*128kB (UM) 14*256kB (UM) 7*512kB (UM) 4*1024kB (M) 4*2048kB (M) 0*4096kB = 36952kB
Node 0 Normal: 1137*4kB (UME) 698*8kB (UME) 308*16kB (UME) 71*32kB (UME) 8*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 17844kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
41 total pagecache pages
1 pages in swap cache
Free swap = 3754960kB
Total swap = 3809276kB
1048442 pages RAM
0 pages HighMem/MovableOnly
113141 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 60KB 183KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 112KB 370KB
vvp_thread_kmem 15KB 15KB
vvp_session_kmem 7KB 7KB
ll_thread_kmem 7KB 7KB
lov_session_kmem 15KB 15KB
lov_thread_kmem 7KB 7KB
osc_session_kmem 7KB 7KB
osc_thread_kmem 61KB 61KB
tgt_fmd_cache 7KB 7KB
tgt_session_kmem 32KB 54KB
tgt_thread_kmem 181KB 181KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 315KB 384KB
ldlm_locks 659KB 880KB
ldlm_resources 328KB 392KB
lnet_msg 87KB 87KB
lnet_rspt 2392KB 2392KB
lnet_small_MDs 31KB 31KB
lnet_MEs 51KB 51KB
ptlrpc_cache 750KB 802KB
cl_env_kmem 3KB 3KB
ll_obd_dev_cache 293KB 293KB
bio-184 31KB 31KB
bio-264 82KB 105KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 7KB 7KB
zil_zcw_cache 7KB 7KB
zil_lwb_cache 15KB 15KB
dbuf_dirty_record_t 9102KB 9102KB
dmu_buf_impl_t 10010KB 10021KB
arc_buf_t 767KB 768KB
zio_buf_comb_16384 4224KB 4736KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 5856KB 5856KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 168KB 288KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 27KB 43KB
zio_cache 457KB 926KB
zfs_btree_leaf_cache 4512KB 4512KB
nfs_commit_data 15KB 15KB
nfs_write_data 47KB 47KB
nfs_read_data 31KB 31KB
nfs_page 8KB 8KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-144 7KB 7KB
fib6_nodes 8KB 8KB
ip6_dst_cache 7KB 7KB
PINGv6 31KB 31KB
RAWv6 31KB 31KB
UDPv6 63KB 63KB
TCPv6 59KB 59KB
mqueue_inode_cache 16KB 16KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
posix_timers_cache 7KB 7KB
UNIX-STREAM 156KB 195KB
UNIX 75KB 75KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 7KB 7KB
ip_fib_alias 7KB 7KB
ip_dst_cache 48KB 48KB
PING 31KB 31KB
RAW 15KB 15KB
UDP 30KB 30KB
tw_sock_TCP 7KB 7KB
request_sock_TCP 7KB 7KB
TCP 162KB 270KB
hugetlbfs_inode_cache 15KB 15KB
bio-272 18KB 18KB
ep_head 8KB 8KB
eventpoll_pwq 24KB 24KB
eventpoll_epi 40KB 40KB
inotify_inode_mark 15KB 15KB
sgpool-128 128KB 128KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 12KB 12KB
bio_integrity_data 3KB 3KB
request_queue 31KB 31KB
blkdev_ioc 7KB 7KB
bio-208 60KB 60KB
biovec-max 320KB 320KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
user_namespace 7KB 7KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_fclone_cache 72KB 72KB
skbuff_head_cache 40KB 40KB
configfs_dir_cache 7KB 7KB
file_lock_cache 7KB 7KB
file_lock_ctx 15KB 15KB
fsnotify_mark_connector 8KB 8KB
taskstats 15KB 15KB
proc_dir_entry 189KB 189KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1031KB 1131KB
kernfs_iattrs_cache 15KB 15KB
kernfs_node_cache 4433KB 4480KB
mnt_cache 213KB 213KB
filp 264KB 312KB
names_cache 160KB 224KB
net_namespace 30KB 30KB
lsm_inode_cache 622KB 836KB
lsm_file_cache 27KB 32KB
key_jar 24KB 24KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 173KB 223KB
vm_area_struct 860KB 860KB
fs_cache 20KB 20KB
files_cache 126KB 126KB
signal_cache 657KB 663KB
sighand_cache 1144KB 1144KB
task_struct 4961KB 5023KB
cred_jar 128KB 133KB
anon_vma_chain 145KB 160KB
anon_vma 158KB 174KB
pid 112KB 114KB
Acpi-Operand 122KB 122KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 23KB 23KB
Acpi-Namespace 15KB 15KB
numa_policy 3KB 3KB
perf_event 30KB 30KB
trace_event_file 193KB 193KB
ftrace_event_field 270KB 270KB
pool_workqueue 112KB 112KB
maple_node 133KB 236KB
task_group 79KB 79KB
mm_struct 189KB 189KB
vmap_area 440KB 594KB
kmalloc-cg-4k 128KB 128KB
kmalloc-cg-2k 512KB 512KB
kmalloc-cg-1k 98KB 128KB
kmalloc-cg-512 48KB 48KB
kmalloc-cg-256 8KB 8KB
kmalloc-cg-192 39KB 39KB
kmalloc-cg-128 8KB 8KB
kmalloc-cg-96 7KB 7KB
kmalloc-cg-64 8KB 8KB
kmalloc-cg-32 12KB 12KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 16KB 16KB
kmalloc-8k 5880KB 6752KB
kmalloc-4k 14464KB 14464KB
kmalloc-2k 2394KB 2432KB
kmalloc-1k 8713KB 8784KB
kmalloc-512 4263KB 5472KB
kmalloc-256 679KB 820KB
kmalloc-192 5293KB 5347KB
kmalloc-128 507KB 812KB
kmalloc-96 1060KB 1519KB
kmalloc-64 1293KB 1376KB
kmalloc-32 394KB 556KB
kmalloc-16 152KB 156KB
kmalloc-8 55KB 56KB
kmem_cache_node 28KB 28KB
kmem_cache 96KB 96KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 514] 0 514 11471 85 106496 512 -250 systemd-journal
[ 529] 0 529 9086 46 90112 448 -1000 systemd-udevd
[ 579] 32 579 3708 64 69632 192 0 rpcbind
[ 580] 0 580 4963 67 53248 224 -1000 auditd
[ 610] 81 610 2684 64 61440 128 -900 dbus-broker-lau
[ 611] 81 611 1262 64 53248 64 -900 dbus-broker
[ 612] 0 612 65187 0 147456 768 0 NetworkManager
[ 615] 0 615 20687 64 61440 64 0 irqbalance
[ 617] 0 617 7312 64 81920 320 0 systemd-logind
[ 620] 997 620 21450 47 73728 160 0 chronyd
[ 638] 0 638 20267 64 57344 0 0 rhsmcertd
[ 640] 0 640 14170 23 81920 256 0 gssproxy
[ 641] 0 641 2974 32 61440 288 -1000 sshd
[ 655] 996 655 20093 69 73728 320 0 munged
[ 681] 0 681 45311 105 98304 320 0 rsyslogd
[ 684] 29 684 2857 41 53248 192 0 rpc.statd
[ 735] 0 735 85090 67 122880 480 0 automount
[ 746] 0 746 766 32 40960 32 0 agetty
[ 747] 0 747 1408 32 49152 32 0 agetty
[ 757] 0 757 2166 64 57344 192 0 crond
[ 1227] 0 1227 4454 64 77824 480 0 sshd-session
[ 1231] 0 1231 5694 64 86016 448 100 systemd
[ 1233] 0 1233 26906 49 90112 896 100 (sd-pam)
[ 1240] 0 1240 4519 64 77824 576 0 sshd-session
[1565572] 0 1565572 4453 64 81920 480 0 sshd-session
[1565575] 0 1565575 4586 72 81920 608 0 sshd-session
[1582502] 0 1582502 1819 64 53248 64 0 bash
[1582535] 0 1582535 1819 32 49152 64 0 bash
[1582536] 0 1582536 2114 32 57344 608 0 tclsh
Link to test
sanityn test 60: Verify data_version behaviour
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 0 PID: 612 Comm: NetworkManager Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.26.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
vpanic+0x129/0x2f0
panic+0x58/0x60
out_of_memory+0x2ae/0x330
__alloc_pages_slowpath.constprop.0+0xa58/0xc60
? node_get_allowed_targets+0xd0/0xf0
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
? __blkcg_rstat_flush.isra.0+0x39/0x180
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x408/0xac0
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
? srso_alias_return_thunk+0x5/0xfbef5
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x1fe/0x690
exc_page_fault+0x74/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0010:do_sys_poll+0x1b8/0x240
Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 70a: cd directory && rm directory ======== 07:37:23 \(1784619443\)
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154709915
obd_memory max: 166862367, obd_memory current: 154721803
obd_memory max: 166862367, obd_memory current: 154721803
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
obd_memory max: 166862367, obd_memory current: 154745179
NetworkManager invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 0 PID: 612 Comm: NetworkManager Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.26.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
dump_header+0x49/0x240
out_of_memory+0x289/0x330
__alloc_pages_slowpath.constprop.0+0xa58/0xc60
? node_get_allowed_targets+0xd0/0xf0
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
? __blkcg_rstat_flush.isra.0+0x39/0x180
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x408/0xac0
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
? srso_alias_return_thunk+0x5/0xfbef5
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x1fe/0x690
exc_page_fault+0x74/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0010:do_sys_poll+0x1b8/0x240
Code: 00 48 85 c0 75 46 49 39 d6 77 41 0f 01 cb 0f ae e8 48 8d 74 24 20 8b 56 08 48 8d 46 0c 85 d2 74 1b 49 8d 4c d5 00 0f b7 50 06 <66> 41 89 55 06 49 83 c5 08 48 83 c0 08 49 39 cd 75 ea 48 8b 36 48
RSP: 0018:ff8dad0700833a70 EFLAGS: 00050202
RAX: ff8dad0700833a9c RBX: ff8dad0700833a9c RCX: 00005584c90ffcb0
RDX: 0000000000000000 RSI: ff8dad0700833a90 RDI: ff4435ed433cdc00
RBP: ff8dad0700833e40 R08: ff8dad0700833da8 R09: ff4435ed41c8c768
R10: 00000000000001d9 R11: 0000000000032208 R12: 0000000000000001
R13: 00005584c90ffc70 R14: 00005584c90ffcb0 R15: ff8dad0700833a90
? srso_alias_return_thunk+0x5/0xfbef5
? __x64_sys_epoll_wait+0x60/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0xff/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? __pfx_pollwake+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? __handle_mm_fault+0x2fb/0x650
__x64_sys_poll+0x39/0x140
? syscall_trace_enter.constprop.0+0x122/0x1a0
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x138/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0xff/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? arch_exit_to_user_mode_prepare.constprop.0+0x78/0x80
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f57bfb032cf
Code: Unable to access opcode bytes at 0x7f57bfb032a5.
RSP: 002b:00007fff6f8c9330 EFLAGS: 00000293 ORIG_RAX: 0000000000000007
RAX: ffffffffffffffda RBX: 00007f57c04ae071 RCX: 00007f57bfb032cf
RDX: 00000000ffffffff RSI: 0000000000000008 RDI: 00005584c90ffc70
RBP: 00005584c90ffc70 R08: 0000000000000000 R09: 00007fff6f8c91c0
R10: 00007fff6f973080 R11: 0000000000000293 R12: 0000000000000008
R13: 0000000000000008 R14: 00007fff6f8c93a0 R15: 00005584c9048000
</TASK>
Mem-Info:
active_anon:0 inactive_anon:9 isolated_anon:0
Node 0 active_anon:0kB inactive_anon:36kB active_file:44kB inactive_file:104kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:40kB dirty:0kB writeback:24kB shmem:4kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:9968kB pagetables:1968kB sec_pagetables:0kB all_unreclaimable? no
Node 0 DMA free:14660kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1626 3598 3598 3598
Node 0 DMA32 free:42064kB boost:0kB min:30416kB low:38020kB high:45624kB reserved_highatomic:4096KB active_anon:0kB inactive_anon:80kB active_file:264kB inactive_file:0kB unevictable:0kB writepending:0kB present:2080624kB managed:1706264kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1972 1972 1972
Node 0 Normal free:24684kB boost:0kB min:36884kB low:46104kB high:55324kB reserved_highatomic:2048KB active_anon:36kB inactive_anon:420kB active_file:0kB inactive_file:216kB unevictable:0kB writepending:24kB present:2097152kB managed:2019572kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 1*4kB (U) 0*8kB 0*16kB 0*32kB 1*64kB (U) 0*128kB 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 3*4096kB (ME) = 14660kB
Node 0 DMA32: 518*4kB (UME) 532*8kB (UME) 482*16kB (UME) 432*32kB (UME) 156*64kB (UME) 29*128kB (UME) 3*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42328kB
Node 0 Normal: 1572*4kB (UME) 740*8kB (UME) 449*16kB (UME) 105*32kB (UME) 25*64kB (UME) 7*128kB (UME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 25248kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
66 total pagecache pages
18 pages in swap cache
Free swap = 3760104kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
113143 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 93KB 247KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 143KB 478KB
vvp_thread_kmem 15KB 15KB
vvp_session_kmem 7KB 7KB
ll_thread_kmem 7KB 7KB
lov_session_kmem 15KB 15KB
lov_thread_kmem 7KB 7KB
osc_session_kmem 7KB 7KB
osc_thread_kmem 61KB 61KB
tgt_fmd_cache 39KB 55KB
tgt_session_kmem 66KB 66KB
tgt_thread_kmem 226KB 226KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 281KB 492KB
ldlm_locks 550KB 1024KB
ldlm_resources 261KB 496KB
lnet_msg 63KB 63KB
lnet_rspt 280KB 280KB
lnet_small_MDs 35KB 35KB
lnet_MEs 44KB 51KB
ptlrpc_cache 725KB 725KB
cl_env_kmem 3KB 3KB
ll_obd_dev_cache 320KB 320KB
bio-184 31KB 31KB
bio-264 61KB 82KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 23KB 23KB
dbuf_dirty_record_t 8900KB 8900KB
dmu_buf_impl_t 10712KB 10776KB
arc_buf_t 754KB 760KB
zio_buf_comb_16384 4960KB 5600KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 5768KB 6976KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 224KB 320KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 207KB 219KB
zio_cache 4995KB 5341KB
zfs_btree_leaf_cache 4620KB 5280KB
nfs_commit_data 15KB 15KB
nfs_write_data 47KB 47KB
nfs_read_data 31KB 31KB
nfs_page 8KB 8KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-144 7KB 7KB
fib6_nodes 4KB 4KB
ip6_dst_cache 7KB 7KB
PINGv6 31KB 31KB
RAWv6 31KB 31KB
UDPv6 63KB 63KB
TCPv6 59KB 59KB
mqueue_inode_cache 16KB 16KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
posix_timers_cache 7KB 7KB
UNIX-STREAM 148KB 195KB
UNIX 75KB 75KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 3KB 3KB
ip_fib_alias 3KB 3KB
ip_dst_cache 40KB 40KB
PING 31KB 31KB
RAW 15KB 15KB
UDP 30KB 30KB
tw_sock_TCP 7KB 7KB
request_sock_TCP 7KB 7KB
TCP 120KB 120KB
hugetlbfs_inode_cache 7KB 7KB
bio-272 15KB 15KB
ep_head 8KB 8KB
eventpoll_pwq 16KB 16KB
eventpoll_epi 32KB 32KB
inotify_inode_mark 15KB 15KB
sgpool-128 160KB 160KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 8KB 8KB
bio_integrity_data 3KB 3KB
request_queue 31KB 31KB
blkdev_ioc 7KB 7KB
bio-208 52KB 52KB
biovec-max 320KB 320KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
user_namespace 7KB 7KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_fclone_cache 88KB 88KB
skbuff_head_cache 36KB 36KB
configfs_dir_cache 3KB 3KB
file_lock_cache 7KB 7KB
file_lock_ctx 7KB 7KB
fsnotify_mark_connector 8KB 8KB
taskstats 15KB 15KB
proc_dir_entry 216KB 216KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1000KB 1039KB
kernfs_iattrs_cache 15KB 15KB
kernfs_node_cache 4449KB 4476KB
mnt_cache 210KB 228KB
filp 263KB 332KB
names_cache 128KB 128KB
net_namespace 30KB 30KB
lsm_inode_cache 618KB 832KB
lsm_file_cache 24KB 32KB
key_jar 24KB 24KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 184KB 215KB
vm_area_struct 953KB 980KB
fs_cache 16KB 16KB
files_cache 110KB 110KB
signal_cache 744KB 771KB
sighand_cache 1282KB 1330KB
task_struct 5551KB 5736KB
cred_jar 143KB 157KB
anon_vma_chain 159KB 196KB
anon_vma 178KB 186KB
pid 130KB 137KB
Acpi-Operand 122KB 122KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 23KB 23KB
Acpi-Namespace 15KB 15KB
numa_policy 3KB 3KB
perf_event 30KB 30KB
trace_event_file 193KB 193KB
ftrace_event_field 270KB 270KB
pool_workqueue 96KB 96KB
maple_node 135KB 248KB
task_group 79KB 79KB
mm_struct 189KB 189KB
vmap_area 460KB 519KB
kmalloc-cg-4k 128KB 128KB
kmalloc-cg-2k 512KB 512KB
kmalloc-cg-1k 94KB 128KB
kmalloc-cg-512 32KB 32KB
kmalloc-cg-256 12KB 12KB
kmalloc-cg-192 31KB 31KB
kmalloc-cg-128 8KB 8KB
kmalloc-cg-96 7KB 7KB
kmalloc-cg-64 8KB 8KB
kmalloc-cg-32 8KB 8KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 15KB 16KB
kmalloc-8k 6528KB 7584KB
kmalloc-4k 14560KB 14560KB
kmalloc-2k 2774KB 2880KB
kmalloc-1k 9214KB 9632KB
kmalloc-512 2994KB 4816KB
kmalloc-256 1119KB 1164KB
kmalloc-192 2623KB 2736KB
kmalloc-128 647KB 912KB
kmalloc-96 983KB 1732KB
kmalloc-64 1333KB 1696KB
kmalloc-32 406KB 708KB
kmalloc-16 146KB 152KB
kmalloc-8 54KB 56KB
kmem_cache_node 24KB 24KB
kmem_cache 88KB 88KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 514] 0 514 7174 96 90112 320 -250 systemd-journal
[ 529] 0 529 9185 64 90112 544 -1000 systemd-udevd
[ 579] 32 579 3708 32 65536 224 0 rpcbind
[ 580] 0 580 4963 41 57344 224 -1000 auditd
[ 610] 81 610 2684 64 61440 160 -900 dbus-broker-lau
[ 611] 81 611 1262 64 49152 64 -900 dbus-broker
[ 612] 0 612 65215 64 147456 736 0 NetworkManager
[ 615] 0 615 20687 32 57344 96 0 irqbalance
[ 618] 0 618 5262 64 77824 320 0 systemd-logind
[ 623] 997 623 21450 72 65536 128 0 chronyd
[ 635] 0 635 20267 32 57344 32 0 rhsmcertd
[ 637] 0 637 2974 64 73728 256 -1000 sshd
[ 647] 996 647 20093 69 77824 320 0 munged
[ 648] 0 648 14170 57 81920 256 0 gssproxy
[ 665] 0 665 41279 96 94208 448 0 rsyslogd
[ 666] 29 666 2857 71 53248 192 0 rpc.statd
[ 726] 0 726 85090 82 122880 512 0 automount
[ 730] 0 730 766 32 49152 32 0 agetty
[ 735] 0 735 1408 32 49152 32 0 agetty
[ 778] 0 778 2166 32 57344 192 0 crond
[ 1217] 0 1217 4454 64 73728 480 0 sshd-session
[ 1221] 0 1221 5693 64 81920 448 100 systemd
[ 1223] 0 1223 26903 81 94208 864 100 (sd-pam)
[ 1230] 0 1230 4519 64 73728 544 0 sshd-session
[ 73142] 0 73142 4453 64 73728 480 0 sshd-session
[ 73145] 0 73145 4518 61 73728 544 0 sshd-session
[ 179132] 0 179132 4453 64 77824 480 0 sshd-session
[ 179135] 0 179135 4551 77 73728 576 0 sshd-session
[ 195622] 0 195622 1052 64 53248 32 0 bash
[ 195624] 0 195624 1052 75 53248 32 0 bash
Link to test
sanityn test 70b: remove files after calling rm_entry
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 1 PID: 178949 Comm: sshd-session Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.17.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
vpanic+0x129/0x2f0
panic+0x58/0x60
out_of_memory+0x2ae/0x330
__alloc_pages_slowpath.constprop.0+0xa70/0xc90
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x24d/0xa50
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x152/0x6a0
exc_page_fault+0x71/0x140
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x55d89a30870d
obd_memory max: 175452651, obd_memory current: 156981627
obd_memory max: 175452651, obd_memory current: 156981627
sshd-session invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 1 PID: 178949 Comm: sshd-session Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.17.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
dump_header+0x49/0x240
out_of_memory+0x289/0x330
__alloc_pages_slowpath.constprop.0+0xa70/0xc90
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x24d/0xa50
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x152/0x6a0
exc_page_fault+0x71/0x140
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x55d89a30870d
Code: Unable to access opcode bytes at 0x55d89a3086e3.
RSP: 002b:00007ffe838cd188 EFLAGS: 00010206
RAX: 0000000000000000 RBX: 000055d89c2d6c40 RCX: 0000000000000017
RDX: 000055d89a2d31d0 RSI: 000055d89c2d6c40 RDI: 000055d89c2752f0
RBP: 000055d89c2752f0 R08: 0000000000000048 R09: 0000000000000001
R10: 00007ffe83915080 R11: 00007ffe838cb188 R12: 0000000000000050
R13: 000055d89c26ae90 R14: 000055d89c265600 R15: 000055d89c2752f0
</TASK>
Mem-Info:
active_anon:0 inactive_anon:1 isolated_anon:0
Node 0 active_anon:0kB inactive_anon:4kB active_file:0kB inactive_file:164kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:72kB dirty:0kB writeback:0kB shmem:4kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:10032kB pagetables:2012kB sec_pagetables:0kB all_unreclaimable? yes
Node 0 DMA free:14636kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1626 3598 3598 3598
Node 0 DMA32 free:37132kB boost:0kB min:30416kB low:38020kB high:45624kB reserved_highatomic:0KB active_anon:80kB inactive_anon:72kB active_file:0kB inactive_file:348kB unevictable:0kB writepending:0kB present:2080624kB managed:1706276kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1972 1972 1972
Node 0 Normal free:36692kB boost:0kB min:36884kB low:46104kB high:55324kB reserved_highatomic:0KB active_anon:216kB inactive_anon:300kB active_file:0kB inactive_file:388kB unevictable:0kB writepending:0kB present:2097152kB managed:2019572kB mlocked:0kB bounce:0kB free_pcp:448kB local_pcp:248kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 1*4kB (U) 1*8kB (E) 2*16kB (UE) 2*32kB (UE) 1*64kB (E) 1*128kB (E) 2*256kB (UE) 1*512kB (E) 1*1024kB (E) 2*2048kB (ME) 2*4096kB (M) = 14636kB
Node 0 DMA32: 1064*4kB (UME) 558*8kB (UME) 490*16kB (UME) 347*32kB (UME) 134*64kB (UME) 5*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 36880kB
Node 0 Normal: 873*4kB (UME) 436*8kB (ME) 423*16kB (UME) 279*32kB (UME) 165*64kB (UME) 5*128kB (UM) 3*256kB (U) 2*512kB (U) 1*1024kB (U) 0*2048kB 0*4096kB = 36692kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
42 total pagecache pages
9 pages in swap cache
Free swap = 3760104kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
113140 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 72KB 294KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 162KB 524KB
vvp_thread_kmem 15KB 15KB
vvp_session_kmem 7KB 7KB
ll_thread_kmem 7KB 7KB
lov_session_kmem 15KB 15KB
lov_thread_kmem 7KB 7KB
osc_session_kmem 7KB 7KB
osc_thread_kmem 61KB 61KB
tgt_fmd_cache 23KB 39KB
tgt_session_kmem 54KB 54KB
tgt_thread_kmem 226KB 226KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 251KB 492KB
ldlm_locks 538KB 1000KB
ldlm_resources 259KB 496KB
lnet_msg 63KB 63KB
lnet_rspt 2008KB 2008KB
lnet_small_MDs 31KB 31KB
lnet_MEs 48KB 51KB
ptlrpc_cache 723KB 771KB
cl_env_kmem 3KB 3KB
ll_obd_dev_cache 347KB 347KB
bio-184 27KB 27KB
bio-264 78KB 78KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 7KB 7KB
dbuf_dirty_record_t 9179KB 9219KB
dmu_buf_impl_t 10199KB 10199KB
arc_buf_t 767KB 768KB
zio_buf_comb_16384 4960KB 5248KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 5568KB 5920KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 160KB 256KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 27KB 43KB
zio_cache 274KB 324KB
zfs_btree_leaf_cache 5168KB 5504KB
nfs_commit_data 15KB 15KB
nfs_write_data 47KB 47KB
nfs_read_data 31KB 31KB
nfs_page 8KB 8KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-144 11KB 11KB
fib6_nodes 8KB 8KB
ip6_dst_cache 7KB 7KB
PINGv6 15KB 15KB
RAWv6 31KB 31KB
UDPv6 63KB 63KB
TCPv6 59KB 59KB
mqueue_inode_cache 16KB 16KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
posix_timers_cache 7KB 7KB
UNIX-STREAM 195KB 195KB
UNIX 75KB 75KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 7KB 7KB
ip_fib_alias 7KB 7KB
ip_dst_cache 40KB 40KB
PING 15KB 15KB
RAW 15KB 15KB
UDP 30KB 30KB
tw_sock_TCP 7KB 7KB
request_sock_TCP 7KB 7KB
TCP 120KB 120KB
hugetlbfs_inode_cache 7KB 7KB
bio-272 15KB 15KB
ep_head 8KB 8KB
eventpoll_pwq 16KB 16KB
eventpoll_epi 32KB 32KB
inotify_inode_mark 15KB 15KB
sgpool-128 96KB 96KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 8KB 8KB
bio_integrity_data 3KB 3KB
request_queue 15KB 15KB
blkdev_ioc 7KB 7KB
bio-208 40KB 40KB
biovec-max 320KB 320KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
user_namespace 7KB 7KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_fclone_cache 80KB 80KB
skbuff_head_cache 32KB 32KB
configfs_dir_cache 7KB 7KB
file_lock_cache 7KB 7KB
file_lock_ctx 7KB 7KB
fsnotify_mark_connector 8KB 8KB
taskstats 15KB 15KB
proc_dir_entry 212KB 212KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1005KB 1054KB
kernfs_iattrs_cache 15KB 15KB
kernfs_node_cache 4444KB 4465KB
mnt_cache 230KB 232KB
filp 278KB 340KB
names_cache 128KB 128KB
net_namespace 30KB 30KB
lsm_inode_cache 624KB 824KB
lsm_file_cache 24KB 32KB
key_jar 28KB 28KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 189KB 215KB
vm_area_struct 964KB 988KB
fs_cache 20KB 20KB
files_cache 126KB 126KB
signal_cache 745KB 771KB
sighand_cache 1284KB 1330KB
task_struct 5587KB 5710KB
cred_jar 137KB 153KB
anon_vma_chain 166KB 180KB
anon_vma 176KB 186KB
pid 129KB 133KB
Acpi-Operand 118KB 118KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 15KB 15KB
Acpi-Namespace 19KB 19KB
numa_policy 3KB 3KB
perf_event 30KB 30KB
trace_event_file 189KB 189KB
ftrace_event_field 270KB 270KB
pool_workqueue 96KB 96KB
maple_node 128KB 240KB
task_group 94KB 94KB
mm_struct 189KB 189KB
vmap_area 457KB 523KB
kmalloc-cg-4k 128KB 128KB
kmalloc-cg-2k 512KB 512KB
kmalloc-cg-1k 91KB 128KB
kmalloc-cg-512 32KB 32KB
kmalloc-cg-256 8KB 8KB
kmalloc-cg-192 31KB 31KB
kmalloc-cg-128 8KB 8KB
kmalloc-cg-96 7KB 7KB
kmalloc-cg-64 8KB 8KB
kmalloc-cg-32 12KB 12KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 16KB 16KB
kmalloc-8k 6576KB 6976KB
kmalloc-4k 14720KB 14720KB
kmalloc-2k 2712KB 2848KB
kmalloc-1k 9467KB 9760KB
kmalloc-512 3011KB 3688KB
kmalloc-256 623KB 680KB
kmalloc-192 2624KB 2732KB
kmalloc-128 406KB 808KB
kmalloc-96 985KB 1598KB
kmalloc-64 1342KB 1532KB
kmalloc-32 410KB 628KB
kmalloc-16 145KB 152KB
kmalloc-8 55KB 56KB
kmem_cache_node 24KB 24KB
kmem_cache 88KB 88KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 514] 0 514 7141 64 94208 320 -250 systemd-journal
[ 529] 0 529 9122 32 98304 512 -1000 systemd-udevd
[ 580] 32 580 3708 32 65536 224 0 rpcbind
[ 581] 0 581 4963 35 53248 224 -1000 auditd
[ 613] 81 613 2684 32 57344 160 -900 dbus-broker-lau
[ 614] 81 614 1262 32 53248 96 -900 dbus-broker
[ 615] 0 615 65213 64 147456 768 0 NetworkManager
[ 618] 0 618 20687 32 57344 96 0 irqbalance
[ 620] 0 620 5261 32 81920 320 0 systemd-logind
[ 622] 997 622 21450 42 69632 160 0 chronyd
[ 638] 0 638 14170 56 86016 256 0 gssproxy
[ 639] 0 639 2974 32 65536 288 -1000 sshd
[ 642] 0 642 20267 32 53248 32 0 rhsmcertd
[ 652] 996 652 20093 69 73728 288 0 munged
[ 682] 0 682 41279 96 81920 256 0 rsyslogd
[ 683] 29 683 2857 71 61440 192 0 rpc.statd
[ 725] 0 725 85090 25 118784 512 0 automount
[ 734] 0 734 766 32 45056 32 0 agetty
[ 735] 0 735 1408 32 53248 32 0 agetty
[ 754] 0 754 2166 32 57344 192 0 crond
[ 1215] 0 1215 4454 64 81920 480 0 sshd-session
[ 1219] 0 1219 5693 32 90112 480 100 systemd
[ 1221] 0 1221 43290 43 98304 896 100 (sd-pam)
[ 1228] 0 1228 4519 63 81920 576 0 sshd-session
[ 73191] 0 73191 4453 64 69632 480 0 sshd-session
[ 73194] 0 73194 4518 62 69632 544 0 sshd-session
[ 178946] 0 178946 4453 64 69632 480 0 sshd-session
[ 178949] 0 178949 4551 43 69632 576 0 sshd-session
[ 195816] 0 195816 1819 64 49152 64 0 bash
[ 195859] 0 195859 1819 54 40960 64 0 bash
[ 195860] 0 195860 1458 64 53248 32 0 dmesg
Link to test
sanityn test 71d: fiemap corruption test with fm_extent_count=0
Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
CPU: 0 PID: 682 Comm: in:imjournal Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.17.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
vpanic+0x129/0x2f0
panic+0x58/0x60
out_of_memory+0x2ae/0x330
__alloc_pages_slowpath.constprop.0+0xa70/0xc90
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
? srso_alias_return_thunk+0x5/0xfbef5
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? pick_next_task_fair+0x1e5/0x510
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x24d/0xa50
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x6a0
exc_page_fault+0x71/0x140
asm_exc_page_fault+0x22/0x30
RIP: 0010:__get_user_8+0x16/0x30
Lustre: DEBUG MARKER: /usr/sbin/lctl mark SKIP: sanityn test_71d ORI-366\/LU-1941: FIEMAP unimplemented on ZFS
Lustre: DEBUG MARKER: SKIP: sanityn test_71d ORI-366/LU-1941: FIEMAP unimplemented on ZFS
obd_memory max: 151381626, obd_memory current: 148510426
obd_memory max: 151381626, obd_memory current: 148522618
obd_memory max: 151381626, obd_memory current: 148522618
obd_memory max: 151381626, obd_memory current: 148522618
in:imjournal invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0
CPU: 0 PID: 682 Comm: in:imjournal Kdump: loaded Tainted: P OE ------ --- 5.14.0-687.17.1.el9_8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x33/0x50
dump_header+0x49/0x240
out_of_memory+0x289/0x330
__alloc_pages_slowpath.constprop.0+0xa70/0xc90
__alloc_pages+0x21c/0x250
alloc_pages_mpol+0x93/0x1f0
? srso_alias_return_thunk+0x5/0xfbef5
__read_swap_cache_async+0x12e/0x280
swap_cluster_readahead+0x1f3/0x2f0
? pick_next_task_fair+0x1e5/0x510
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? filemap_get_entry+0xe3/0x140
swapin_readahead+0xce/0xe0
do_swap_page+0x24d/0xa50
? __pte_offset_map+0x17/0x160
? srso_alias_return_thunk+0x5/0xfbef5
__handle_mm_fault+0x2fb/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x6a0
exc_page_fault+0x71/0x140
asm_exc_page_fault+0x22/0x30
RIP: 0010:__get_user_8+0x16/0x30
Code: 00 0f 1f 00 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 90 48 ba 00 f0 ff ff ff ff ff 00 48 39 c2 48 19 d2 48 09 d0 0f 01 cb <48> 8b 10 31 c0 0f 01 ca e9 f8 85 7d 00 66 66 2e 0f 1f 84 00 00 00
RSP: 0018:ff8923d5809a3e50 EFLAGS: 00050206
RAX: 00007efdc35c2fe8 RBX: ff4556fec1b1c680 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ff8923d5809a3e80 RDI: ff4556fec1b1c680
RBP: ff8923d5809a3ef0 R08: 0000000000000100 R09: 0000000000000012
R10: ff4556fec333a70c R11: ff4556fec25bce88 R12: 0000000000000002
R13: 00007efdc47d1e31 R14: ff8923d5809a3f58 R15: 0000000000000000
rseq_get_rseq_cs+0x1d/0x230
rseq_ip_fixup+0x45/0x1a0
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x138/0x270
__rseq_handle_notify_resume+0x25/0xc0
exit_to_user_mode_loop+0xd8/0x130
exit_to_user_mode_prepare+0x6b/0x80
irqentry_exit_to_user_mode+0x5/0x30
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7efdc47d1e31
Code: Unable to access opcode bytes at 0x7efdc47d1e07.
RSP: 002b:00007efdc35c1920 EFLAGS: 00010206
RAX: 0000000000000000 RBX: 00000000000dbba0 RCX: 00007efdc45033ce
RDX: 0000000000000000 RSI: 0000000000000001 RDI: 00005608afca4b90
RBP: 00005608afca4b90 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000293 R12: 00007efdc35c1ad0
R13: 00007efdc35c1b98 R14: 00007efdc35c1ba0 R15: 00007efdbc01c960
</TASK>
Mem-Info:
active_anon:0 inactive_anon:0 isolated_anon:0
Node 0 active_anon:0kB inactive_anon:0kB active_file:152kB inactive_file:36kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4kB dirty:0kB writeback:0kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:8816kB pagetables:1832kB sec_pagetables:0kB all_unreclaimable? yes
Node 0 DMA free:14652kB boost:0kB min:280kB low:348kB high:416kB reserved_highatomic:0KB active_anon:12kB inactive_anon:4kB active_file:108kB inactive_file:16kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 1678 3598 3598 3598
Node 0 DMA32 free:41084kB boost:0kB min:31388kB low:39232kB high:47076kB reserved_highatomic:2048KB active_anon:12kB inactive_anon:0kB active_file:224kB inactive_file:560kB unevictable:0kB writepending:0kB present:2080624kB managed:1752944kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB
lowmem_reserve[]: 0 0 1920 1920 1920
Node 0 Normal free:18804kB boost:0kB min:35912kB low:44888kB high:53864kB reserved_highatomic:2048KB active_anon:0kB inactive_anon:4kB active_file:0kB inactive_file:276kB unevictable:0kB writepending:0kB present:2097152kB managed:1972912kB mlocked:0kB bounce:0kB free_pcp:280kB local_pcp:280kB free_cma:0kB
lowmem_reserve[]: 0 0 0 0 0
Node 0 DMA: 4*4kB (ME) 2*8kB (UM) 2*16kB (UM) 3*32kB (UME) 3*64kB (UME) 2*128kB (UE) 3*256kB (UME) 2*512kB (ME) 2*1024kB (ME) 1*2048kB (U) 2*4096kB (ME) = 14688kB
Node 0 DMA32: 480*4kB (UME) 435*8kB (UME) 370*16kB (UME) 296*32kB (UME) 190*64kB (UME) 47*128kB (UME) 0*256kB 1*512kB (U) 0*1024kB 1*2048kB (U) 0*4096kB = 41528kB
Node 0 Normal: 662*4kB (UME) 351*8kB (UME) 179*16kB (UME) 62*32kB (ME) 4*64kB (M) 0*128kB 0*256kB 0*512kB 0*1024kB 2*2048kB (U) 1*4096kB (U) = 18752kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
80 total pagecache pages
0 pages in swap cache
Free swap = 3755000kB
Total swap = 3807228kB
1048442 pages RAM
0 pages HighMem/MovableOnly
113138 pages reserved
0 pages cma reserved
0 pages hwpoisoned
Unreclaimable slab info:
Name Used Total
ofd_obj 53KB 131KB
osd_zapit_cache 15KB 15KB
zfs_osd_obj 113KB 239KB
upd_kmem 3KB 3KB
lqe_kmem 35KB 35KB
tgt_fmd_cache 7KB 7KB
tgt_session_kmem 50KB 50KB
tgt_thread_kmem 181KB 181KB
ldlm_glimpse_work_kmem 7KB 7KB
interval_tree 105KB 244KB
ldlm_locks 181KB 576KB
ldlm_resources 89KB 248KB
lnet_msg 63KB 63KB
lnet_rspt 1240KB 1240KB
lnet_small_MDs 16KB 31KB
lnet_MEs 49KB 51KB
ptlrpc_cache 764KB 818KB
ll_obd_dev_cache 213KB 213KB
bio-184 35KB 35KB
bio-264 82KB 97KB
zfs_znode_hold_cache 7KB 7KB
zap_attr_long_cache 30KB 30KB
zap_attr_cache 7KB 7KB
zap_name 7KB 7KB
zil_zcw_cache 7KB 7KB
zil_lwb_cache 15KB 15KB
dbuf_dirty_record_t 8842KB 9044KB
dmu_buf_impl_t 9454KB 10199KB
arc_buf_t 746KB 784KB
zio_buf_comb_16384 3600KB 3712KB
zio_buf_comb_12288 48KB 48KB
zio_buf_comb_8192 4504KB 5088KB
zio_buf_comb_6144 60KB 60KB
zio_buf_comb_4096 152KB 256KB
zio_buf_comb_3072 60KB 60KB
zio_buf_comb_2048 64KB 64KB
zio_buf_comb_1536 63KB 63KB
zio_link_cache 48KB 167KB
zio_cache 852KB 3056KB
zfs_btree_leaf_cache 3300KB 4096KB
nfs_commit_data 15KB 15KB
nfs_write_data 47KB 47KB
nfs_read_data 31KB 31KB
nfs_page 8KB 8KB
rpc_buffers 64KB 64KB
rpc_tasks 8KB 8KB
ext4_system_zone 3KB 3KB
ext4_io_end_vec 8KB 8KB
jbd2_inode 8KB 8KB
scsi_sense_cache 24KB 24KB
bio-144 23KB 23KB
fib6_nodes 8KB 8KB
ip6_dst_cache 7KB 7KB
PINGv6 31KB 31KB
RAWv6 31KB 31KB
UDPv6 63KB 63KB
TCPv6 59KB 59KB
mqueue_inode_cache 16KB 16KB
kioctx 15KB 15KB
aio_kiocb 7KB 7KB
posix_timers_cache 7KB 7KB
UNIX-STREAM 120KB 180KB
UNIX 75KB 75KB
tcp_bind_bucket 8KB 8KB
ip_fib_trie 7KB 7KB
ip_fib_alias 7KB 7KB
ip_dst_cache 48KB 48KB
PING 31KB 31KB
RAW 15KB 15KB
UDP 30KB 30KB
tw_sock_TCP 7KB 7KB
request_sock_TCP 7KB 7KB
TCP 192KB 300KB
hugetlbfs_inode_cache 7KB 7KB
bio-272 15KB 15KB
ep_head 8KB 8KB
eventpoll_pwq 24KB 24KB
eventpoll_epi 48KB 48KB
inotify_inode_mark 15KB 15KB
sgpool-128 160KB 256KB
sgpool-64 64KB 64KB
sgpool-32 32KB 32KB
sgpool-16 16KB 16KB
sgpool-8 8KB 8KB
bio_integrity_data 3KB 3KB
request_queue 31KB 31KB
blkdev_ioc 7KB 7KB
bio-208 40KB 56KB
biovec-max 288KB 352KB
biovec-128 64KB 64KB
biovec-64 32KB 32KB
biovec-16 8KB 8KB
khugepaged_mm_slot 7KB 7KB
user_namespace 7KB 7KB
uid_cache 7KB 7KB
dmaengine-unmap-256 30KB 30KB
dmaengine-unmap-128 15KB 15KB
dmaengine-unmap-16 3KB 3KB
dmaengine-unmap-2 4KB 4KB
audit_buffer 7KB 7KB
skbuff_fclone_cache 32KB 72KB
skbuff_head_cache 48KB 48KB
configfs_dir_cache 3KB 3KB
file_lock_cache 7KB 7KB
file_lock_ctx 19KB 19KB
fsnotify_mark_connector 8KB 8KB
taskstats 15KB 15KB
proc_dir_entry 192KB 192KB
pde_opener 7KB 7KB
seq_file 7KB 7KB
sigqueue 7KB 7KB
shmem_inode_cache 1002KB 1100KB
kernfs_iattrs_cache 15KB 15KB
kernfs_node_cache 4173KB 4213KB
mnt_cache 202KB 202KB
filp 256KB 316KB
names_cache 192KB 192KB
net_namespace 30KB 30KB
lsm_inode_cache 607KB 844KB
lsm_file_cache 23KB 32KB
key_jar 24KB 24KB
uts_namespace 15KB 15KB
nsproxy 7KB 7KB
vma_lock 170KB 199KB
vm_area_struct 872KB 872KB
fs_cache 20KB 20KB
files_cache 112KB 126KB
signal_cache 663KB 663KB
sighand_cache 1144KB 1144KB
task_struct 5023KB 5023KB
cred_jar 129KB 133KB
anon_vma_chain 137KB 160KB
anon_vma 171KB 174KB
pid 110KB 114KB
Acpi-Operand 122KB 122KB
Acpi-ParseExt 7KB 7KB
Acpi-Parse 7KB 7KB
Acpi-State 15KB 15KB
Acpi-Namespace 15KB 15KB
numa_policy 3KB 3KB
perf_event 30KB 30KB
trace_event_file 193KB 193KB
ftrace_event_field 270KB 270KB
pool_workqueue 120KB 120KB
maple_node 117KB 216KB
task_group 79KB 79KB
mm_struct 221KB 221KB
vmap_area 397KB 535KB
kmalloc-cg-4k 128KB 128KB
kmalloc-cg-2k 490KB 544KB
kmalloc-cg-1k 90KB 128KB
kmalloc-cg-512 48KB 48KB
kmalloc-cg-256 12KB 12KB
kmalloc-cg-192 39KB 39KB
kmalloc-cg-128 8KB 8KB
kmalloc-cg-96 7KB 7KB
kmalloc-cg-64 8KB 8KB
kmalloc-cg-32 12KB 12KB
kmalloc-cg-16 8KB 8KB
kmalloc-cg-8 16KB 16KB
kmalloc-8k 5376KB 5728KB
kmalloc-4k 14436KB 14468KB
kmalloc-2k 2262KB 2336KB
kmalloc-1k 7826KB 8352KB
kmalloc-512 3661KB 4512KB
kmalloc-256 720KB 1008KB
kmalloc-192 5247KB 5272KB
kmalloc-128 547KB 828KB
kmalloc-96 932KB 1527KB
kmalloc-64 1172KB 1408KB
kmalloc-32 366KB 592KB
kmalloc-16 141KB 148KB
kmalloc-8 55KB 56KB
kmem_cache_node 24KB 24KB
kmem_cache 96KB 96KB
Tasks state (memory values in pages):
[ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name
[ 513] 0 513 11405 32 98304 512 -250 systemd-journal
[ 526] 0 526 9185 81 90112 544 -1000 systemd-udevd
[ 578] 32 578 3708 64 69632 192 0 rpcbind
[ 579] 0 579 4963 35 53248 224 -1000 auditd
[ 609] 81 609 2684 64 57344 128 -900 dbus-broker-lau
[ 610] 81 610 1262 64 45056 64 -900 dbus-broker
[ 611] 0 611 65218 32 147456 768 0 NetworkManager
[ 614] 0 614 20687 32 61440 96 0 irqbalance
[ 616] 0 616 7311 64 81920 320 0 systemd-logind
[ 619] 997 619 21450 49 73728 160 0 chronyd
[ 636] 0 636 14170 50 86016 256 0 gssproxy
[ 638] 0 638 2973 64 61440 256 -1000 sshd
[ 639] 0 639 20267 32 57344 64 0 rhsmcertd
[ 642] 996 642 20093 69 69632 320 0 munged
[ 668] 29 668 2857 76 57344 160 0 rpc.statd
[ 669] 0 669 45311 24 102400 544 0 rsyslogd
[ 723] 0 723 85088 43 131072 512 0 automount
[ 732] 0 732 766 64 40960 0 0 agetty
[ 733] 0 733 1408 32 53248 32 0 agetty
[ 757] 0 757 2166 32 49152 192 0 crond
[ 1214] 0 1214 4454 64 73728 480 0 sshd-session
[ 1218] 0 1218 5693 64 81920 480 100 systemd
[ 1220] 0 1220 26905 79 94208 864 100 (sd-pam)
[ 1227] 0 1227 4519 41 73728 544 0 sshd-session
[1380075] 0 1380075 4453 32 69632 512 0 sshd-session
[1380078] 0 1380078 4551 32 77824 608 0 sshd-session
[1397103] 0 1397103 1052 64 45056 64 0 bash
[1397124] 0 1397124 1052 1 45056 64 0 bash
Link to test
Return to new crashes list