| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| sanity-benchmark test bonnie: bonnie++ | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 0 PID: 30453 Comm: mmp Kdump: loaded Tainted: P OE --------- - - 4.18.0-513.9.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac out_of_memory.cold.36+0x2f/0x7e __alloc_pages_slowpath+0xbf0/0xcd0 __alloc_pages_nodemask+0x2e2/0x330 new_slab+0x3ed/0x520 ___slab_alloc.part.91+0x441/0x740 ? spl_kmem_cache_alloc+0x5d/0x160 [spl] ? __switch_to_asm+0x43/0x80 ? finish_task_switch+0x86/0x2f0 ? spl_kmem_cache_alloc+0x5d/0x160 [spl] kmem_cache_alloc+0x252/0x280 spl_kmem_cache_alloc+0x5d/0x160 [spl] zio_create+0x3e/0x4b0 [zfs] zio_null+0x2b/0x40 [zfs] mmp_write_uberblock+0x29a/0x710 [zfs] mmp_thread+0x3e9/0x740 [zfs] ? mmp_write_uberblock+0x710/0x710 [zfs] ? __thread_exit+0x20/0x20 [spl] thread_generic_wrapper+0x5a/0x70 [spl] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 7446528kB available, using 5419832kB file size Lustre: DEBUG MARKER: min OST has 7446528kB available, using 5419832kB file size obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 144165459 obd_memory max: 316475111, obd_memory current: 145254639 obd_memory max: 316475111, obd_memory current: 145256079 obd_memory max: 316475111, obd_memory current: 145256079 obd_memory max: 316475111, obd_memory current: 145264263 obd_memory max: 316475111, obd_memory current: 145281743 obd_memory max: 316475111, obd_memory current: 145267183 obd_memory max: 316475111, obd_memory current: 145242167 obd_memory max: 316475111, obd_memory current: 145242447 obd_memory max: 316475111, obd_memory current: 145291295 obd_memory max: 316475111, obd_memory current: 145291295 obd_memory max: 316475111, obd_memory current: 145291295 obd_memory max: 316475111, obd_memory current: 145327183 obd_memory max: 316475111, obd_memory current: 145346823 obd_memory max: 316475111, obd_memory current: 145342655 obd_memory max: 316475111, obd_memory current: 145309799 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145305055 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145188207 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145200415 obd_memory max: 316475111, obd_memory current: 145205647 obd_memory max: 316475111, obd_memory current: 145222159 obd_memory max: 316475111, obd_memory current: 145224319 obd_memory max: 316475111, obd_memory current: 145214831 obd_memory max: 316475111, obd_memory current: 145214831 mmp invoked oom-killer: gfp_mask=0x6042c0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=0, oom_score_adj=0 CPU: 0 PID: 30453 Comm: mmp Kdump: loaded Tainted: P OE --------- - - 4.18.0-513.9.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 dump_header+0x4a/0x1df out_of_memory.cold.36+0xa/0x7e __alloc_pages_slowpath+0xbf0/0xcd0 __alloc_pages_nodemask+0x2e2/0x330 new_slab+0x3ed/0x520 ___slab_alloc.part.91+0x441/0x740 ? spl_kmem_cache_alloc+0x5d/0x160 [spl] ? __switch_to_asm+0x43/0x80 ? finish_task_switch+0x86/0x2f0 ? spl_kmem_cache_alloc+0x5d/0x160 [spl] kmem_cache_alloc+0x252/0x280 spl_kmem_cache_alloc+0x5d/0x160 [spl] zio_create+0x3e/0x4b0 [zfs] zio_null+0x2b/0x40 [zfs] mmp_write_uberblock+0x29a/0x710 [zfs] mmp_thread+0x3e9/0x740 [zfs] ? mmp_write_uberblock+0x710/0x710 [zfs] ? __thread_exit+0x20/0x20 [spl] thread_generic_wrapper+0x5a/0x70 [spl] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Mem-Info: active_anon:81 inactive_anon:292 isolated_anon:67 Node 0 active_anon:148kB inactive_anon:1296kB active_file:24kB inactive_file:128kB unevictable:0kB isolated(anon):268kB isolated(file):0kB mapped:1036kB dirty:0kB writeback:652kB shmem:52kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB kernel_stack:9536kB pagetables:6264kB all_unreclaimable? no Node 0 DMA free:10512kB min:256kB low:320kB high:384kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 2597 2597 2597 2597 Node 0 DMA32 free:61348kB min:65276kB low:76472kB high:87668kB active_anon:504kB inactive_anon:1076kB active_file:40kB inactive_file:4kB unevictable:0kB writepending:500kB present:3129320kB managed:2694556kB mlocked:0kB bounce:0kB free_pcp:8kB local_pcp:8kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 0*4kB 0*8kB 1*16kB (U) 0*32kB 0*64kB 0*128kB 1*256kB (U) 0*512kB 0*1024kB 1*2048kB (M) 2*4096kB (ME) = 10512kB Node 0 DMA32: 475*4kB (UME) 403*8kB (UME) 328*16kB (UME) 224*32kB (UMEH) 212*64kB (UME) 147*128kB (UMH) 42*256kB (UM) 2*512kB (UM) 0*1024kB 0*2048kB 0*4096kB = 61700kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 212 total pagecache pages 163 pages in swap cache Swap cache stats: add 24451, delete 24275, find 200/274 Free swap = 2767660kB Total swap = 2841596kB 786328 pages RAM 0 pages HighMem/MovableOnly 108849 pages reserved 0 pages hwpoisoned Unreclaimable slab info: Name Used Total ofd_obj 102KB 172KB osd_zapit_cache 15KB 15KB zfs_osd_obj 162KB 324KB upd_kmem 7KB 7KB lqe_kmem 24KB 24KB vvp_thread_kmem 15KB 15KB vvp_session_kmem 7KB 7KB ll_thread_kmem 7KB 7KB lov_session_kmem 15KB 15KB lov_thread_kmem 7KB 7KB osc_session_kmem 7KB 7KB osc_thread_kmem 90KB 90KB ldlm_glimpse_work_kmem 7KB 7KB interval_tree 116KB 132KB interval_node 116KB 132KB ldlm_locks 127KB 173KB ldlm_resources 86KB 98KB lnet_msg 100KB 100KB lnet_rspt 167KB 167KB lnet_small_MDs 131KB 131KB lnet_MEs 134KB 134KB ptlrpc_cache 549KB 614KB tgt_fmd_cache 79KB 79KB tgt_session_kmem 110KB 110KB tgt_thread_kmem 246KB 246KB cl_env_kmem 3KB 3KB ll_obd_dev_cache 252KB 252KB zfs_znode_hold_cache 7KB 7KB zfs_znode_cache 31KB 31KB dmu_buf_impl_t 3354KB 3646KB arc_buf_t 268KB 278KB arc_buf_hdr_t_full 8716KB 17025KB dnode_t 9362KB 11516KB sa_cache 95KB 172KB abd_t 308KB 1311KB lz4_cache 64KB 64KB zio_buf_comb_16384 15936KB 18688KB zio_buf_comb_12288 48KB 48KB zio_buf_comb_10240 48KB 48KB zio_buf_comb_8192 192KB 192KB zio_buf_comb_7168 64KB 64KB zio_buf_comb_6144 64KB 64KB zio_buf_comb_5120 64KB 64KB zio_buf_comb_4096 288KB 544KB zio_buf_comb_3584 63KB 63KB zio_buf_comb_3072 60KB 60KB zio_buf_comb_2560 60KB 60KB zio_buf_comb_2048 64KB 64KB zio_buf_comb_1536 63KB 63KB zio_buf_comb_1024 300KB 800KB zio_buf_comb_512 260KB 1328KB zio_link_cache 123KB 123KB zio_cache 2060KB 2238KB zfs_btree_leaf_cache 3116KB 4032KB mod_hash_entries 3KB 3KB bio-200 44KB 44KB bio-272 131KB 135KB nfs_commit_data 15KB 15KB nfs_write_data 47KB 47KB nfs_read_data 31KB 31KB nfs_page 8KB 8KB fscache_cookie_jar 3KB 3KB rpc_buffers 64KB 64KB rpc_tasks 8KB 8KB ext4_system_zone 3KB 3KB jbd2_inode 40KB 40KB scsi_sense_cache 4KB 4KB bio-160 11KB 11KB fib6_nodes 8KB 8KB ip6_dst_cache 7KB 7KB RAWv6 15KB 15KB UDPv6 63KB 63KB TCPv6 61KB 61KB sgpool-128 32KB 32KB sgpool-64 32KB 32KB sgpool-32 16KB 16KB sgpool-16 8KB 8KB sgpool-8 4KB 4KB mqueue_inode_cache 16KB 16KB kioctx 31KB 31KB aio_kiocb 7KB 7KB bio-248 8KB 8KB posix_timers_cache 7KB 7KB UNIX 346KB 346KB tcp_bind_bucket 8KB 8KB ip_fib_trie 7KB 7KB ip_fib_alias 7KB 7KB ip_dst_cache 68KB 68KB RAW 15KB 15KB UDP 30KB 30KB tw_sock_TCP 7KB 7KB request_sock_TCP 7KB 7KB TCP 95KB 95KB hugetlbfs_inode_cache 15KB 15KB bio-280 11KB 11KB eventpoll_pwq 19KB 19KB eventpoll_epi 36KB 36KB inotify_inode_mark 7KB 7KB request_queue 63KB 63KB blkdev_ioc 31KB 31KB bio-224 80KB 80KB biovec-max 544KB 704KB biovec-128 64KB 64KB biovec-64 32KB 32KB biovec-16 8KB 8KB bio_integrity_payload 4KB 4KB khugepaged_mm_slot 7KB 7KB uid_cache 7KB 7KB dmaengine-unmap-256 30KB 30KB dmaengine-unmap-128 15KB 15KB dmaengine-unmap-16 3KB 3KB dmaengine-unmap-2 4KB 4KB audit_buffer 7KB 7KB skbuff_fclone_cache 102KB 102KB skbuff_head_cache 68KB 68KB file_lock_cache 7KB 7KB file_lock_ctx 7KB 7KB fsnotify_mark_connector 7KB 7KB net_namespace 31KB 31KB task_delay_info 59KB 59KB taskstats 15KB 15KB proc_dir_entry 248KB 248KB pde_opener 7KB 7KB seq_file 8KB 8KB sigqueue 7KB 7KB shmem_inode_cache 875KB 875KB kernfs_iattrs_cache 15KB 15KB kernfs_node_cache 4389KB 4412KB mnt_cache 291KB 291KB filp 400KB 520KB names_cache 64KB 64KB avc_node 7KB 7KB lsm_inode_cache 1033KB 1544KB lsm_file_cache 68KB 68KB key_jar 24KB 24KB nsproxy 7KB 7KB vm_area_struct 1222KB 1232KB mm_struct 220KB 220KB fs_cache 16KB 16KB files_cache 110KB 110KB signal_cache 733KB 750KB sighand_cache 1198KB 1206KB task_struct 6030KB 6090KB cred_jar 170KB 185KB anon_vma_chain 239KB 252KB anon_vma 235KB 248KB pid 96KB 96KB Acpi-Operand 74KB 74KB Acpi-ParseExt 3KB 3KB Acpi-Parse 3KB 3KB Acpi-State 11KB 11KB Acpi-Namespace 15KB 15KB numa_policy 3KB 3KB perf_event 31KB 31KB trace_event_file 154KB 154KB ftrace_event_field 211KB 211KB pool_workqueue 12KB 12KB task_group 157KB 157KB vmap_area 248KB 360KB kmalloc-cg-4k 96KB 96KB kmalloc-cg-2k 96KB 96KB kmalloc-cg-1k 191KB 224KB kmalloc-cg-512 16KB 16KB kmalloc-cg-256 8KB 8KB kmalloc-cg-192 19KB 19KB kmalloc-cg-96 7KB 7KB kmalloc-cg-64 8KB 8KB kmalloc-cg-32 8KB 8KB kmalloc-cg-16 8KB 8KB kmalloc-cg-8 8KB 8KB kmalloc-8k 6384KB 7072KB kmalloc-4k 5456KB 5792KB kmalloc-2k 3902KB 4000KB kmalloc-1k 10628KB 11056KB kmalloc-512 6282KB 7408KB kmalloc-256 394KB 480KB kmalloc-192 2209KB 2390KB kmalloc-128 574KB 996KB kmalloc-96 1182KB 1519KB kmalloc-64 1110KB 1536KB kmalloc-32 1337KB 2184KB kmalloc-16 144KB 144KB kmalloc-8 48KB 48KB kmem_cache_node 24KB 24KB kmem_cache 149KB 149KB Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss pgtables_bytes swapents oom_score_adj name [ 485] 0 485 22392 1 212992 258 0 systemd-journal [ 520] 0 520 24224 1 204800 299 -1000 systemd-udevd [ 580] 32 580 16813 0 167936 187 0 rpcbind [ 583] 0 583 14277 0 131072 160 -1000 auditd [ 609] 0 609 31228 0 139264 163 0 irqbalance [ 611] 0 611 20397 1 192512 242 0 systemd-logind [ 614] 70 614 14604 0 163840 180 0 avahi-daemon [ 615] 81 615 14142 1 159744 227 -900 dbus-daemon [ 618] 0 618 95190 0 360448 1181 0 NetworkManager [ 621] 995 621 35020 0 180224 190 0 chronyd [ 622] 70 622 14538 0 151552 110 0 avahi-daemon [ 633] 0 633 19163 4 200704 232 -1000 sshd [ 635] 0 635 30232 0 262144 861 0 cupsd [ 638] 0 638 121260 51 442368 4556 0 tuned [ 642] 994 642 29757 0 155648 171 0 munged [ 645] 0 645 28884 4 118784 136 0 rhsmcertd [ 675] 0 675 52381 0 180224 1327 0 rsyslogd [ 684] 29 684 11292 0 122880 141 0 rpc.statd [ 709] 0 709 12230 7 126976 137 0 xinetd [ 905] 998 905 402491 0 319488 1587 0 polkitd [ 949] 0 949 26834 0 188416 250 0 master [ 950] 0 950 26275 1 192512 169 0 gssproxy [ 952] 89 952 30855 1 221184 267 0 qmgr [ 1018] 0 1018 104372 0 307200 967 0 automount [ 1021] 0 1021 6465 10 102400 209 0 crond [ 1022] 0 1022 6533 1 98304 48 0 atd [ 1044] 0 1044 2371 0 65536 36 0 agetty [ 1049] 0 1049 1729 0 57344 33 0 agetty [ 91064] 89 91064 30842 0 217088 269 0 pickup [ 96796] 0 96796 31620 8 290816 308 0 sshd [ 96800] 0 96800 22397 13 200704 345 0 systemd [ 96802] 0 96802 54796 1 294912 760 0 (sd-pam) [ 96808] 0 96808 31620 13 286720 308 0 sshd | Link to test |