| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanityn test 70a: cd directory | Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled CPU: 1 UID: 0 PID: 1932877 Comm: ll_ost00_042 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.47.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary) Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 vpanic+0xc2/0x290 panic+0x6c/0x6c out_of_memory.cold+0x61/0x78 __alloc_pages_slowpath.constprop.0+0x59b/0xb80 __alloc_pages_noprof+0x31d/0x330 alloc_pages_mpol_noprof+0x8d/0x1d0 allocate_slab+0x275/0x310 ? srso_alias_return_thunk+0x5/0xfbef5 ? spl_kmem_cache_alloc+0x59/0x160 [spl] ___slab_alloc+0x65b/0xb60 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? __try_to_del_timer_sync+0x58/0xa0 ? spl_kmem_cache_alloc+0x59/0x160 [spl] ? spl_kmem_cache_alloc+0x59/0x160 [spl] ? srso_alias_return_thunk+0x5/0xfbef5 ? dbuf_read+0x4b0/0x6b0 [zfs] ? spl_kmem_cache_alloc+0x59/0x160 [spl] kmem_cache_alloc_noprof+0xe3/0x3d0 spl_kmem_cache_alloc+0x59/0x160 [spl] dbuf_create+0x3d/0x2e0 [zfs] dbuf_hold_impl+0x3e9/0x720 [zfs] dbuf_hold_level+0x30/0x60 [zfs] dnode_free_range+0x4f3/0x740 [zfs] dmu_free_long_range_impl+0x32c/0x480 [zfs] dmu_free_long_range+0x7a/0xc0 [zfs] osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs] osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs] osd_trans_stop+0x3d9/0x530 [osd_zfs] ofd_destroy+0x38a/0x720 [ofd] ofd_destroy_by_fid+0x309/0x470 [ofd] ofd_destroy_hdl+0x137/0x990 [ofd] tgt_handle_request0.isra.0+0x124/0x810 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfbef5 tgt_request_handle+0x232/0xd50 [ptlrpc] ptlrpc_server_handle_request.isra.0+0x293/0xda0 [ptlrpc] ? lprocfs_counter_add+0x12a/0x190 [obdclass] ptlrpc_main+0xaeb/0xed0 [ptlrpc] ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Crash core archive creation started: kernel-crash.trevis-153vm258.vmcore.tar.gz Crash core archive created: kernel-crash.trevis-153vm258.vmcore.tar.gz | Lustre: DEBUG MARKER: /usr/sbin/lctl mark == sanityn test 70b: remove files after calling rm_entry ========================================================== 07:22:25 \(1788592945\) obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 obd_memory max: 151324715, obd_memory current: 139197461 ll_ost00_042 invoked oom-killer: gfp_mask=0x42cc0(GFP_KERNEL|__GFP_NOWARN|__GFP_COMP), order=0, oom_score_adj=0 CPU: 1 UID: 0 PID: 1932877 Comm: ll_ost00_042 Kdump: loaded Tainted: P OE ------ --- 6.12.0-211.47.1.el10_2_lustre.x86_64 #1 PREEMPT(voluntary) Tainted: [P]=PROPRIETARY_MODULE, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 dump_header+0x43/0x1ad out_of_memory.cold+0x3e/0x78 __alloc_pages_slowpath.constprop.0+0x59b/0xb80 __alloc_pages_noprof+0x31d/0x330 alloc_pages_mpol_noprof+0x8d/0x1d0 allocate_slab+0x275/0x310 ? srso_alias_return_thunk+0x5/0xfbef5 ? spl_kmem_cache_alloc+0x59/0x160 [spl] ___slab_alloc+0x65b/0xb60 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? __try_to_del_timer_sync+0x58/0xa0 ? spl_kmem_cache_alloc+0x59/0x160 [spl] ? spl_kmem_cache_alloc+0x59/0x160 [spl] ? srso_alias_return_thunk+0x5/0xfbef5 ? dbuf_read+0x4b0/0x6b0 [zfs] ? spl_kmem_cache_alloc+0x59/0x160 [spl] kmem_cache_alloc_noprof+0xe3/0x3d0 spl_kmem_cache_alloc+0x59/0x160 [spl] dbuf_create+0x3d/0x2e0 [zfs] dbuf_hold_impl+0x3e9/0x720 [zfs] dbuf_hold_level+0x30/0x60 [zfs] dnode_free_range+0x4f3/0x740 [zfs] dmu_free_long_range_impl+0x32c/0x480 [zfs] dmu_free_long_range+0x7a/0xc0 [zfs] osd_unlinked_object_free+0x4a/0x3c0 [osd_zfs] osd_unlinked_list_emptify+0xb0/0xc0 [osd_zfs] osd_trans_stop+0x3d9/0x530 [osd_zfs] ofd_destroy+0x38a/0x720 [ofd] ofd_destroy_by_fid+0x309/0x470 [ofd] ofd_destroy_hdl+0x137/0x990 [ofd] tgt_handle_request0.isra.0+0x124/0x810 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfbef5 tgt_request_handle+0x232/0xd50 [ptlrpc] ptlrpc_server_handle_request.isra.0+0x293/0xda0 [ptlrpc] ? lprocfs_counter_add+0x12a/0x190 [obdclass] ptlrpc_main+0xaeb/0xed0 [ptlrpc] ? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> Mem-Info: active_anon:9 inactive_anon:0 isolated_anon:0 Node 0 active_anon:36kB inactive_anon:0kB active_file:84kB inactive_file:148kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:8kB dirty:0kB writeback:0kB shmem:0kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:0kB writeback_tmp:0kB kernel_stack:8768kB pagetables:2356kB sec_pagetables:0kB all_unreclaimable? no Balloon:0kB Node 0 DMA free:14536kB boost:0kB min:284kB low:352kB high:420kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15360kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 1647 3572 3572 3572 Node 0 DMA32 free:38320kB boost:0kB min:30688kB low:38360kB high:46032kB reserved_highatomic:0KB active_anon:36kB inactive_anon:0kB active_file:16kB inactive_file:48kB unevictable:0kB writepending:0kB present:2080624kB managed:1687408kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 1924 1924 1924 Node 0 Normal free:18240kB boost:0kB min:36608kB low:45760kB high:54912kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:8kB inactive_file:160kB unevictable:0kB writepending:0kB present:2097152kB managed:1970496kB mlocked:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 0 Node 0 DMA: 5*4kB (UE) 3*8kB (UE) 2*16kB (UE) 2*32kB (UE) 1*64kB (E) 2*128kB (UE) 1*256kB (E) 1*512kB (E) 1*1024kB (E) 2*2048kB (ME) 2*4096kB (M) = 14540kB Node 0 DMA32: 482*4kB (UME) 471*8kB (UME) 487*16kB (UME) 346*32kB (UME) 184*64kB (UME) 11*128kB (ME) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37744kB Node 0 Normal: 770*4kB (UME) 375*8kB (UME) 314*16kB (UME) 138*32kB (UME) 41*64kB (UME) 2*128kB (U) 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 18400kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 54 total pagecache pages 9 pages in swap cache Free swap = 3743728kB Total swap = 3805180kB 1048442 pages RAM 0 pages HighMem/MovableOnly 130126 pages reserved 0 pages cma reserved 0 pages hwpoisoned Unreclaimable slab info: Name Used Total ofd_obj 71KB 239KB osd_zapit_cache 15KB 15KB zfs_osd_obj 138KB 393KB upd_kmem 7KB 7KB lqe_kmem 31KB 31KB tgt_fmd_cache 7KB 7KB tgt_session_kmem 66KB 85KB tgt_thread_kmem 166KB 166KB ldlm_glimpse_work_kmem 7KB 7KB interval_tree 342KB 512KB ldlm_locks 703KB 1048KB ldlm_resources 338KB 520KB lnet_msg 55KB 55KB lnet_rspt 412KB 412KB lnet_small_MDs 39KB 39KB lnet_MEs 48KB 51KB ptlrpc_cache 694KB 694KB ll_obd_dev_cache 221KB 221KB bio-168 31KB 31KB bio-248 80KB 80KB zfs_znode_hold_cache 7KB 7KB zap_attr_long_cache 30KB 30KB zap_attr_cache 7KB 7KB zap_name 7KB 7KB zil_zcw_cache 7KB 7KB zil_lwb_cache 15KB 15KB dbuf_dirty_record_t 8716KB 8716KB dmu_buf_impl_t 10068KB 10068KB arc_buf_t 752KB 752KB zio_buf_comb_16384 4336KB 4448KB zio_buf_comb_12288 48KB 48KB zio_buf_comb_8192 3368KB 3584KB zio_buf_comb_6144 60KB 60KB zio_buf_comb_4096 216KB 320KB zio_buf_comb_3072 60KB 60KB zio_buf_comb_2048 64KB 64KB zio_buf_comb_1536 63KB 63KB zio_link_cache 20KB 31KB zio_cache 338KB 400KB zfs_btree_leaf_cache 3260KB 3840KB nfs_commit_data 15KB 15KB nfs_write_data 47KB 47KB nfs_read_data 31KB 31KB nfs_page 8KB 8KB netfs_subrequest 19KB 19KB netfs_request 67KB 67KB AF_VSOCK 15KB 15KB ext4_system_zone 3KB 3KB ext4_io_end_vec 8KB 8KB jbd2_inode 8KB 8KB scsi_sense_cache 24KB 24KB bio-128 11KB 11KB rpc_buffers 64KB 64KB rpc_tasks 8KB 8KB fib6_node 8KB 8KB ip6_dst_cache 8KB 8KB PINGv6 30KB 30KB RAWv6 46KB 46KB UDPv6 31KB 31KB TCPv6 61KB 61KB bio-256 7KB 7KB mqueue_inode_cache 15KB 15KB kioctx 15KB 15KB aio_kiocb 7KB 7KB UNIX-STREAM 141KB 175KB UNIX 79KB 79KB tcp_bind2_bucket 8KB 8KB tcp_bind_bucket 8KB 8KB ip_fib_trie 3KB 3KB ip_fib_alias 3KB 3KB rtable 39KB 39KB PING 32KB 32KB RAW 16KB 16KB UDP 78KB 78KB tw_sock_TCP 8KB 8KB request_sock_TCP 7KB 7KB TCP 173KB 277KB hugetlbfs_inode_cache 15KB 15KB bio-272 11KB 11KB ep_head 8KB 8KB eventpoll_pwq 24KB 24KB eventpoll_epi 54KB 68KB inotify_inode_mark 15KB 15KB sgpool-128 192KB 192KB sgpool-64 64KB 64KB sgpool-32 32KB 32KB sgpool-16 16KB 16KB sgpool-8 8KB 8KB bio_integrity_data 4KB 4KB request_queue 30KB 30KB blkdev_ioc 7KB 7KB bio-192 44KB 60KB biovec-max 288KB 352KB biovec-128 64KB 64KB biovec-64 32KB 32KB biovec-16 8KB 8KB khugepaged_mm_slot 7KB 7KB memdup_user-1k 16KB 16KB memdup_user-256 8KB 8KB memdup_user-128 8KB 8KB memdup_user-64 8KB 8KB memdup_user-32 8KB 8KB memdup_user-16 8KB 8KB memdup_user-8 8KB 8KB memdup_user-192 7KB 7KB memdup_user-96 7KB 7KB user_namespace 15KB 15KB uid_cache 7KB 7KB dmaengine-unmap-256 30KB 30KB dmaengine-unmap-128 15KB 15KB dmaengine-unmap-16 3KB 3KB dmaengine-unmap-2 4KB 4KB audit_buffer 7KB 7KB skbuff_small_head 126KB 142KB skbuff_fclone_cache 64KB 64KB skbuff_head_cache 52KB 52KB configfs_dir_cache 3KB 3KB file_lock_cache 7KB 7KB file_lock_ctx 15KB 15KB fsnotify_mark_connector 7KB 7KB posix_timers_cache 47KB 47KB taskstats 15KB 15KB proc_dir_entry 185KB 189KB pde_opener 7KB 7KB seq_file 7KB 7KB sigqueue 7KB 7KB shmem_inode_cache 1088KB 1209KB kernfs_iattrs_cache 19KB 19KB kernfs_node_cache 4236KB 4263KB mnt_cache 204KB 204KB filp 239KB 259KB names_cache 160KB 224KB net_namespace 62KB 62KB lsm_inode_cache 339KB 856KB lsm_file_cache 25KB 32KB key_jar 28KB 28KB uts_namespace 15KB 15KB nsproxy 7KB 7KB vma_lock 165KB 191KB vm_area_struct 717KB 747KB fs_cache 20KB 20KB files_cache 59KB 110KB signal_cache 608KB 614KB sighand_cache 1113KB 1113KB task_struct 4708KB 4708KB cred 131KB 133KB anon_vma_chain 137KB 156KB anon_vma 198KB 203KB pid 114KB 114KB Acpi-Operand 122KB 122KB Acpi-ParseExt 7KB 7KB Acpi-Parse 7KB 7KB Acpi-State 23KB 23KB Acpi-Namespace 19KB 19KB numa_policy 3KB 3KB perf_event 15KB 15KB trace_event_file 220KB 220KB ftrace_event_field 339KB 339KB pool_workqueue 136KB 136KB maple_node 200KB 328KB task_group 75KB 75KB mm_struct 218KB 218KB vmap_area 399KB 618KB kmalloc_buckets 3KB 3KB kmalloc-cg-4k 240KB 288KB kmalloc-cg-2k 688KB 832KB kmalloc-cg-1k 157KB 192KB kmalloc-cg-512 64KB 64KB kmalloc-cg-256 20KB 20KB kmalloc-cg-128 28KB 44KB kmalloc-cg-64 17KB 40KB kmalloc-cg-32 22KB 28KB kmalloc-cg-16 8KB 8KB kmalloc-cg-8 16KB 16KB kmalloc-cg-192 53KB 70KB kmalloc-cg-96 36KB 59KB kmalloc-8k 5808KB 5888KB kmalloc-4k 13824KB 13824KB kmalloc-2k 2326KB 2336KB kmalloc-1k 7744KB 8032KB kmalloc-512 4604KB 4960KB kmalloc-256 689KB 732KB kmalloc-128 509KB 844KB kmalloc-64 1198KB 1356KB kmalloc-32 402KB 596KB kmalloc-16 161KB 164KB kmalloc-8 49KB 52KB kmalloc-192 5244KB 5343KB kmalloc-96 1122KB 1677KB kmem_cache_node 28KB 28KB kmem_cache 127KB 127KB Tasks state (memory values in pages): [ pid ] uid tgid total_vm rss rss_anon rss_file rss_shmem pgtables_bytes swapents oom_score_adj name [ 660] 0 660 10309 105 32 58 15 86016 352 -250 systemd-journal [ 698] 0 698 3979 62 32 30 0 69632 224 0 systemd-userdbd [ 709] 0 709 8623 73 32 41 0 81920 576 -1000 systemd-udevd [ 769] 32 769 3163 46 32 14 0 61440 224 0 rpcbind [ 770] 0 770 5105 75 47 28 0 57344 192 -1000 auditd [ 802] 81 802 2203 87 32 55 0 57344 128 -900 dbus-broker-lau [ 804] 81 804 1224 71 32 39 0 45056 96 -900 dbus-broker [ 805] 0 805 82715 68 32 36 0 143360 672 0 NetworkManager [ 806] 994 806 21172 93 32 61 0 65536 128 0 chronyd [ 808] 0 808 20106 48 0 48 0 53248 96 0 irqbalance [ 810] 0 810 4195 66 32 34 0 69632 256 0 systemd-logind [ 831] 0 831 30676 79 32 47 0 98304 512 0 gssproxy [ 833] 0 833 2196 58 32 26 0 65536 256 -1000 sshd [ 841] 0 841 20281 64 32 32 0 57344 0 0 rhsmcertd [ 886] 0 886 46729 93 32 32 29 114688 288 0 rsyslogd [ 893] 29 893 2654 97 42 55 0 57344 192 0 rpc.statd [ 902] 996 902 19223 50 10 40 0 69632 288 0 munged [ 945] 0 945 84317 63 32 31 0 118784 480 0 automount [ 958] 0 958 1248 52 0 52 0 45056 32 0 agetty [ 959] 0 959 1890 74 32 42 0 57344 32 0 agetty [ 986] 0 986 2077 12 0 12 0 61440 192 0 crond [ 1306] 0 1306 3795 58 32 26 0 77824 448 0 sshd-session [ 1311] 0 1311 5172 58 32 26 0 73728 512 100 systemd [ 1313] 0 1313 3415 22 9 13 0 61440 288 100 (sd-pam) [ 1322] 0 1322 3885 67 33 34 0 73728 512 0 sshd-session [1906187] 0 1906187 3795 51 32 19 0 65536 448 0 sshd-session [1906190] 0 1906190 3921 54 35 19 0 69632 576 0 sshd-session [1921838] 0 1921838 4092 65 32 33 0 73728 224 0 systemd-userwor [1929823] 0 1929823 4092 57 32 25 0 69632 192 0 systemd-userwor [1929824] 0 1929824 4092 63 32 31 0 73728 192 0 systemd-userwor [1932939] 0 1932939 1773 74 32 42 0 49152 64 0 bash [1932980] 0 1932980 2346 69 32 37 0 57344 960 0 tclsh | Link to test (2026-09-05 04:20) |