| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| sanity-benchmark test iozone: iozone | LustreError: 122116:0:(osc_page.c:183:osc_page_delete()) ASSERTION( 0 ) failed: LustreError: 122116:0:(osc_page.c:183:osc_page_delete()) LBUG Pid: 122116, comm: iozone 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Apr 24 08:27:29 EDT 2025 Call Trace TBD: Kernel panic - not syncing: LBUG CPU: 1 PID: 122116 Comm: iozone Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x18/0x18 [libcfs] osc_page_delete+0x320/0x320 [osc] __cl_page_delete+0x7c/0x210 [obdclass] cl_page_delete+0x2b/0xf0 [obdclass] ll_invalidate_folio+0xfa/0x2e0 [lustre] truncate_cleanup_folio+0x54/0x90 truncate_inode_pages_range+0xea/0x400 ll_truncate_inode_pages_final+0x18/0xe0 [lustre] ll_delete_inode+0x39/0x150 [lustre] evict+0xcc/0x1d0 __dentry_kill+0x71/0x170 dput+0xdc/0x1a0 __fput+0xe3/0x250 task_work_run+0x59/0x90 do_exit+0x264/0x480 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 do_group_exit+0x2d/0x90 __x64_sys_exit_group+0x14/0x20 do_syscall_64+0x5c/0xf0 ? srso_alias_return_thunk+0x5/0xfbef5 ? __count_memcg_events+0x4f/0xb0 ? srso_alias_return_thunk+0x5/0xfbef5 ? mm_account_fault+0x6c/0x100 ? srso_alias_return_thunk+0x5/0xfbef5 ? handle_mm_fault+0x116/0x270 ? srso_alias_return_thunk+0x5/0xfbef5 ? do_user_addr_fault+0x1d6/0x6a0 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? exc_page_fault+0x62/0x150 entry_SYSCALL_64_after_hwframe+0x78/0x80 RIP: 0033:0x7fc5254d931d | Lustre: DEBUG MARKER: /usr/sbin/lctl mark min OST has 7244440kB available, using 7488720kB file size Lustre: DEBUG MARKER: min OST has 7244440kB available, using 7488720kB file size Lustre: 6525:0:(client.c:2289:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1760150096/real 1760150096] req@00000000a48fc006 x1845644996344064/t0(0) o4->lustre-OST0006-osc-ff1b05b4410ff800@10.240.46.197@tcp:6/4 lens 584/472 e 0 to 1 dl 1760150146 ref 2 fl Rpc:XQr/0/ffffffff rc 0/-1 job:'iozone.500' Lustre: lustre-OST0006-osc-ff1b05b4410ff800: Connection to lustre-OST0006 (at 10.240.46.197@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 6525:0:(client.c:2289:ptlrpc_expire_one_request()) @@@ Request sent has timed out for sent delay: [sent 1760150099/real 0] req@00000000e6e44b3d x1845644996363904/t0(0) o4->lustre-OST0003-osc-ff1b05b4410ff800@10.240.46.197@tcp:6/4 lens 600/472 e 0 to 1 dl 1760150150 ref 3 fl Rpc:Xr/0/ffffffff rc 0/-1 job:'iozone.500' Lustre: 6525:0:(client.c:2289:ptlrpc_expire_one_request()) Skipped 17 previous similar messages LustreError: 166-1: MGC10.240.46.198@tcp: Connection to MGS (at 10.240.46.198@tcp) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000-mdc-ff1b05b4410ff800: Connection to lustre-MDT0000 (at 10.240.46.198@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Evicted from MGS (at 10.240.46.198@tcp) after server handle changed from 0xc72be329fb5b4e7b to 0xc72be32a15ae7481 Lustre: Skipped 6 previous similar messages LustreError: lustre-MDT0000-mdc-ff1b05b4410ff800: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. Lustre: MGC10.240.46.198@tcp: Connection restored to 10.240.46.198@tcp (at 10.240.46.198@tcp) Lustre: Skipped 3 previous similar messages LustreError: 122318:0:(file.c:245:ll_close_inode_openhandle()) lustre-clilmv-ff1b05b4410ff800: inode [0x200004292:0x12339:0x0] mdc close failed: rc = -108 LNetError: 6515:0:(lib-socket.c:352:lnet_sock_getaddr()) Error -107 getting sock peer IP/port LNetError: 6515:0:(socklnd_lib.c:44:ksocknal_lib_get_conn_addrs()) Error -107 getting sock peer_ni IP LustreError: 6518:0:(events.c:211:client_bulk_callback()) event type 1, status -5, desc 0000000030bafeaa LustreError: 6522:0:(osc_request.c:1036:osc_init_grant()) lustre-OST0001-osc-ff1b05b4410ff800: granted 8437760 but already consumed 75669504 LustreError: lustre-OST0001-osc-ff1b05b4410ff800: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]// may get corrupted (rc -5) Lustre: 6526:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]// may get corrupted (rc -108) Lustre: 6526:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]// may get corrupted (rc -108) Lustre: 6526:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]// may get corrupted (rc -108) LustreError: lustre-OST0003-osc-ff1b05b4410ff800: This client was evicted by lustre-OST0003; in progress operations using this service will fail. LustreError: lustre-OST0000-osc-ff1b05b4410ff800: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: lustre-OST0004-osc-ff1b05b4410ff800: This client was evicted by lustre-OST0004; in progress operations using this service will fail. Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]// may get corrupted (rc -5) LustreError: lustre-OST0005-osc-ff1b05b4410ff800: This client was evicted by lustre-OST0005; in progress operations using this service will fail. LustreError: lustre-OST0006-osc-ff1b05b4410ff800: This client was evicted by lustre-OST0006; in progress operations using this service will fail. Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]/ may get corrupted (rc -5) Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]/ may get corrupted (rc -5) Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]/ may get corrupted (rc -5) Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]/ may get corrupted (rc -5) Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]/ may get corrupted (rc -5) Lustre: 6523:0:(llite_lib.c:3585:ll_dirty_page_discard_warn()) lustre: dirty page discard: 10.240.46.198@tcp:/lustre/fid: [0x200004292:0x12339:0x0]/ may get corrupted (rc -5) systemd-coredump[122319]: Process 467 (systemd-journal) of user 0 dumped core. systemd-coredump[122319]: Coredump diverted to /var/lib/systemd/coredump/core.systemd-journal.0.d1b1981258c6429282e3ede169df8fe3.467.1760150156000000.zst systemd-coredump[122319]: Stack trace of thread 467: systemd-coredump[122319]: #0 0x00007f992fa7a297 event_get_clock_data.lto_priv.0 (libsystemd-shared-252.so + 0x27a297) systemd-coredump[122319]: #1 0x00007f992fa7fb89 event_source_time_prioq_reshuffle.lto_priv.0 (libsystemd-shared-252.so + 0x27fb89) systemd-coredump[122319]: #2 0x00007f992fa7fdb8 source_set_pending.lto_priv.0 (libsystemd-shared-252.so + 0x27fdb8) systemd-coredump[122319]: #3 0x00007f992fa87398 process_timer (libsystemd-shared-252.so + 0x287398) systemd-coredump[122319]: #4 0x00007f992fa88593 sd_event_wait (libsystemd-shared-252.so + 0x288593) systemd-coredump[122319]: #5 0x00007f992fa89c7b sd_event_run (libsystemd-shared-252.so + 0x289c7b) systemd-coredump[122319]: #6 0x0000559bc6381065 main (systemd-journald + 0xa065) systemd-coredump[122319]: #7 0x00007f992f4295d0 __libc_start_call_main (libc.so.6 + 0x295d0) systemd-coredump[122319]: #8 0x00007f992f429680 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x29680) systemd-coredump[122319]: #9 0x0000559bc6381745 _start (systemd-journald + 0xa745) systemd-coredump[122319]: ELF object binary architecture: AMD x86-64 systemd[1]: systemd-journald.service: Main process exited, code=dumped, status=6/ABRT systemd[1]: systemd-journald.service: Failed with result 'watchdog'. systemd[1]: systemd-journald.service: Consumed 2.343s CPU time. systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 2. systemd[1]: Stopped Journal Service. systemd[1]: systemd-journald.service: Consumed 2.343s CPU time. LustreError: 122116:0:(osc_cache.c:2492:osc_teardown_async_page()) extent 000000001cdc96d2@{[18176 -> 18303/18431], [2|0|+|cache|wiuY|00000000cb39e860], [548864|128|+|-|000000003450ecbc|1024|0000000000000000]} trunc at 18176. LustreError: 122116:0:(osc_cache.c:2492:osc_teardown_async_page()) ### extent: 000000001cdc96d2 ns: lustre-OST0002-osc-ff1b05b4410ff800 lock: 000000003450ecbc/0x423ea6f4d0b96d11 lrc: 82/0,0 mode: PW/PW res: [0x5602b:0x0:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->524287) gid 0 flags: 0x800020000000000 nid: local remote: 0xfed5b840b14004cb expref: -99 pid: 122116 timeout: 0 lvb_type: 1 LustreError: 122116:0:(osc_page.c:181:osc_page_delete()) page@00000000a44b6703[2 000000006e80d360 5 1 0000000000000000] LustreError: 122116:0:(osc_page.c:181:osc_page_delete()) vmpage @0000000039bd80df 17ffffd004201d 3:0 ff1b05b44e613900 128512 lru LustreError: 122116:0:(osc_page.c:181:osc_page_delete()) osc-page@00000000ebc5645f 18176: 1< 0x845fed 2 + - > 2< 74448896 0 4096 0x0 0x420 | 0000000000000000 00000000640fdb06 00000000cb39e860 > 3< 0 0 0 > 4< 0 9 8 443375616 - | + - + - > 5< + - + - | 0 - | 2048 - +> LustreError: 122116:0:(osc_page.c:181:osc_page_delete()) end page@00000000a44b6703 LustreError: 122116:0:(osc_page.c:181:osc_page_delete()) Trying to teardown failed: -16 | Link to test |