| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 | LustreError: 2402:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 2402:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 2402 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? lbug_with_loc.cold.7+0x5/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpc_request_cache_free+0xca/0xe0 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Kernel panic - not syncing: LBUG CPU: 0 PID: 2402 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? panic+0x142/0x44c ? lbug_with_loc.cold.7+0x1a/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpc_request_cache_free+0xca/0xe0 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: 2403:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779473777/real 1779473777] req@ffff9b6c52729180 x1865911886803200/t0(0) o400->MGC192.168.201.114@tcp@192.168.201.114@tcp:26/25 lens 224/224 e 0 to 1 dl 1779473793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 | Link to test |
| replay-dual test 16: fail MDS during recovery (3571) | LustreError: 2365:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 2365:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 1 PID: 2365 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? lbug_with_loc.cold.7+0x5/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Kernel panic - not syncing: LBUG CPU: 1 PID: 2365 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? panic+0x142/0x44c ? lbug_with_loc.cold.7+0x1a/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 31115:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff95a24a0a5000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 LustreError: 31115:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message Lustre: Unmounted lustre-client Lustre: 2369:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779473286/real 1779473286] req@ffff95a248633100 x1865911845289216/t0(0) o400->MGC192.168.202.109@tcp@192.168.202.109@tcp:26/25 lens 224/224 e 0 to 1 dl 1779473302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000-mdc-ffff95a251283000: Connection to lustre-MDT0000 (at 192.168.202.109@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 2369:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 18 previous similar messages Lustre: Skipped 4 previous similar messages LustreError: MGC192.168.202.109@tcp: Connection to MGS (at 192.168.202.109@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 3 previous similar messages Lustre: Evicted from MGS (at 192.168.202.109@tcp) after server handle changed from 0x7132919afdcfe728 to 0x7132919afdd0277f Lustre: Skipped 3 previous similar messages LustreError: lustre-MDT0000-mdc-ffff95a251283000: operation ldlm_enqueue to node 192.168.202.109@tcp failed: rc = -19 LustreError: Skipped 1 previous similar message | Link to test |
| replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 | LustreError: 2406:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 2406:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 3 PID: 2406 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? lbug_with_loc.cold.7+0x5/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Kernel panic - not syncing: LBUG CPU: 3 PID: 2406 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? panic+0x142/0x44c ? lbug_with_loc.cold.7+0x1a/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: 2409:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779447395/real 1779447395] req@ffff8df31ab23800 x1865884196043136/t0(0) o400->MGC192.168.202.132@tcp@192.168.202.132@tcp:26/25 lens 224/224 e 0 to 1 dl 1779447411 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 LustreError: MGC192.168.202.132@tcp: Connection to MGS (at 192.168.202.132@tcp) was lost; in progress operations using this service will fail Lustre: Evicted from MGS (at 192.168.202.132@tcp) after server handle changed from 0x628d3061c0688bc6 to 0x628d3061c0689184 | Link to test |
| replay-dual test 16: fail MDS during recovery (3571) | LustreError: 2364:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 2364:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 2364 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? lbug_with_loc.cold.7+0x5/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 Kernel panic - not syncing: LBUG CPU: 0 PID: 2364 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 Call Trace: ? dump_stack+0xbb/0x10e ? panic+0x142/0x44c ? lbug_with_loc.cold.7+0x1a/0x4e [libcfs] ? ptlrpc_send_new_req+0xe7e/0x1230 [ptlrpc] ? ptlrpc_check_set+0xacd/0x3060 [ptlrpc] ? ptlrpcd+0xa82/0xe40 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x540/0x540 [ptlrpc] ? kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ? ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 LustreError: 31037:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ffff9a5a50233800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 LustreError: 31037:0:(lov_obd.c:792:lov_cleanup()) Skipped 1 previous similar message Lustre: Unmounted lustre-client Lustre: 2366:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1779446827/real 1779446827] req@ffff9a5a45e18000 x1865884195377664/t0(0) o400->MGC192.168.203.123@tcp@192.168.203.123@tcp:26/25 lens 224/224 e 0 to 1 dl 1779446843 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 2366:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 25 previous similar messages LustreError: MGC192.168.203.123@tcp: Connection to MGS (at 192.168.203.123@tcp) was lost; in progress operations using this service will fail LustreError: Skipped 2 previous similar messages Lustre: Evicted from MGS (at 192.168.203.123@tcp) after server handle changed from 0xd68a207427879f69 to 0xd68a20742787a13e Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-mdc-ffff9a5a45660800: Connection to lustre-MDT0000 (at 192.168.203.123@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages | Link to test |