| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| replay-dual test 0a: expired recovery with lost client | LustreError: 2021300:0:(client.c:1861:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 2021300:0:(client.c:1861:ptlrpc_send_new_req()) LBUG CPU: 6 PID: 2021300 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ptlrpc_send_new_req+0xe86/0x1230 [ptlrpc] ptlrpc_check_set+0xa92/0x2ff0 [ptlrpc] ptlrpcd+0xaeb/0xe90 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x550/0x550 [ptlrpc] kthread+0x1de/0x210 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000-osc-ffff903680d63800: disconnect after 21s idle LDISKFS-fs (dm-0): 15 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.8@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect LustreError: lustre-MDT0000-mdc-ffff90361ae29000: operation ldlm_enqueue to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff90361ae29000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Client 1f554f34-ea06-4715-adf5-ed0edbfe8df6 (at 0@lo) reconnected, waiting for 2 clients in recovery for 1:09 | Link to test (2026-09-08 23:45) |
| sanity-pfl test 22d: out of/low on space + failed to repeat + forced extension | LustreError: 1017627:0:(client.c:1861:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 1017627:0:(client.c:1861:ptlrpc_send_new_req()) LBUG CPU: 9 PID: 1017627 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ptlrpc_send_new_req+0xe86/0x1230 [ptlrpc] ptlrpc_check_set+0xa92/0x2ff0 [ptlrpc] ptlrpcd+0xaeb/0xe90 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x550/0x550 [ptlrpc] kthread+0x1de/0x210 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001-osc-ffff8f262355c000: disconnect after 25s idle LDISKFS-fs (dm-0): 6 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.123.45@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect LustreError: lustre-MDT0000-mdc-ffff8f262355c000: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff8f262355c000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: lustre-MDT0000: Client 9c0615e3-3599-4a6c-990c-5e16ee7f5d12 (at 0@lo) reconnected, waiting for 1 clients in recovery for 1:10 | Link to test (2026-09-08 06:17) |
| replay-dual test 0a: expired recovery with lost client | LustreError: 3253109:0:(client.c:1860:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 3253109:0:(client.c:1860:ptlrpc_send_new_req()) LBUG CPU: 15 PID: 3253109 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G W O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ptlrpc_send_new_req+0xe86/0x1230 [ptlrpc] ptlrpc_check_set+0xa92/0x2ff0 [ptlrpc] ptlrpcd+0xaeb/0xe90 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x550/0x550 [ptlrpc] kthread+0x1de/0x210 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 | LustreError: 3258420:0:(osd_handler.c:720:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001-osc-ffff8f48c69f8000: disconnect after 20s idle LustreError: MGC192.168.123.83@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect LustreError: lustre-MDT0000-mdc-ffff8f490eb50000: operation ldlm_enqueue to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-mdc-ffff8f490eb50000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Client 02e2a60c-99c1-4a7f-bb55-17d34e6b3712 (at 0@lo) reconnected, waiting for 2 clients in recovery for 1:10 | Link to test (2026-08-29 17:57) |
| replay-single test 3c: replay failed open -ENOMEM | LustreError: 70222:0:(client.c:1860:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 70222:0:(client.c:1860:ptlrpc_send_new_req()) LBUG CPU: 4 PID: 70222 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G O -------- - - 4.18.0rocky8.10-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.9.0+1408+7b966129 04/01/2014 Call Trace: dump_stack+0x99/0xca lbug_with_loc.cold.4+0xd/0x86 [libcfs] ptlrpc_send_new_req+0xe86/0x1230 [ptlrpc] ptlrpc_check_set+0xa92/0x2ff0 [ptlrpc] ptlrpcd+0xaeb/0xe90 [ptlrpc] ? do_wait_intr+0xf0/0xf0 ? ptlrpcd_add_req+0x550/0x550 [ptlrpc] kthread+0x1de/0x210 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: *** cfs_fail_loc=128, val=0*** Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: 70236:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787729487/real 1787729487] req@ffff9955c9c62d80 x1874569864820736/t0(0) o400->lustre-MDT0000-mdc-ffff9955cfa20800@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787729503 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 70236:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 28 previous similar messages LDISKFS-fs (dm-0): 4 truncates cleaned up LDISKFS-fs (dm-0): recovery complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 3 previous similar messages LustreError: lustre-MDT0000-mdc-ffff9955cfa20800: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 9 previous similar messages Lustre: lustre-MDT0000-mdc-ffff9955cfa20800: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Client 7ee9f142-bdb9-45ff-a987-55e946a5de3f (at 0@lo) reconnected, waiting for 1 clients in recovery for 1:09 | Link to test (2026-08-26 07:31) |
| replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3 | LustreError: 4279:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 4279:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 4279 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc] ptlrpc_check_set+0x7eb/0x2080 [ptlrpc] ptlrpcd+0x7a7/0x9c0 [ptlrpc] ? do_wait_intr_irq+0xa0/0xa0 ? ptlrpcd_add_req+0x350/0x350 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 | Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 | Link to test (2026-05-22 17:23) |
| replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3 | LustreError: 4098:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 4098:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 4098 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc] ptlrpc_check_set+0x7eb/0x2080 [ptlrpc] ptlrpcd+0x7a7/0x9c0 [ptlrpc] ? do_wait_intr_irq+0xa0/0xa0 ? ptlrpcd_add_req+0x350/0x350 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 | Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all | Link to test (2026-05-22 17:18) |
| replay-single test 70f: OSS O_DIRECT recovery with 2 clients | LustreError: 4255:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 4255:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 4255 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc] ptlrpc_check_set+0x7eb/0x2080 [ptlrpc] ptlrpcd+0x7a7/0x9c0 [ptlrpc] ? do_wait_intr_irq+0xa0/0xa0 ? ptlrpcd_add_req+0x350/0x350 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 | Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: DEBUG MARKER: mount | grep /mnt/lustre' ' Autotest: Test running for 75 minutes (lustre-reviews_review-dne-zfs-part-6_125445.39) Lustre: DEBUG MARKER: dd bs=1M count=10 if=/dev/urandom of=/tmp/f70f.replay-single Lustre: DEBUG MARKER: md5sum /tmp/f70f.replay-single Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 1 times Lustre: DEBUG MARKER: test_70f failing OST 1 times Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm284.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid Lustre: DEBUG MARKER: trevis-155vm284.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 2 times Lustre: DEBUG MARKER: test_70f failing OST 2 times | Link to test (2026-05-22 10:59) |
| recovery-mds-scale test failover_ost: failover OST | LustreError: 8006:0:(client.c:1684:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 8006:0:(client.c:1684:ptlrpc_send_new_req()) LBUG Pid: 8006, comm: ptlrpcd_00_00 4.18.0-372.32.1.el8_6.x86_64 #1 SMP Fri Oct 7 12:35:10 EDT 2022 Call Trace TBD: [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] ptlrpc_send_new_req+0x9d5/0xbc0 [ptlrpc] [<0>] ptlrpc_check_set+0xab8/0x1f60 [ptlrpc] [<0>] ptlrpcd+0x6ca/0xa50 [ptlrpc] [<0>] kthread+0x10a/0x120 [<0>] ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bi Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: dd on trevis-84vm8 Lustre: DEBUG MARKER: Started client load: dd on trevis-84vm8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: tar on trevis-84vm9 Lustre: DEBUG MARKER: Started client load: tar on trevis-84vm9 Lustre: DEBUG MARKER: cat /tmp/client-load.pid Lustre: DEBUG MARKER: /usr/sbin/lctl mark ==== Check clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1200 Lustre: DEBUG MARKER: ==== Check clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1200 Lustre: DEBUG MARKER: ps auxwww | grep -v grep | grep -q run_dd.sh Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait ost3 recovery complete before next failover Lustre: DEBUG MARKER: Wait ost3 recovery complete before next failover Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0000.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0001.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0002.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0002.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0003.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0003.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0004.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0004.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0005.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0005.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0006.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0006.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Checking clients are in FULL\|IDLE state before next failover Lustre: DEBUG MARKER: Checking clients are in FULL|IDLE state before next failover Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm9.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid,os Lustre: DEBUG MARKER: trevis-84vm9.trevis.whamcloud.com: executing wait_import_state_mount FULL|IDLE osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0003-osc-[-0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid,os Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid,os Lustre: DEBUG MARKER: trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL|IDLE osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0003-osc-[-0 Lustre: DEBUG MARKER: trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL|IDLE osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0003-osc-[-0 Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark Starting failover on ost3 Lustre: DEBUG MARKER: Starting failover on ost3 LNet: Timeout error while writing to 10.240.40.33:7988. Closing socket: rc = -110 Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1674110310/real 1674110310] req@000000002072c3a5 x1755429199296064/t0(0) o400->lustre-OST0000-osc-ffff904885d98000@10.240.40.33@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110386 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: lustre-OST0000-osc-ffff904885d98000: Connection to lustre-OST0000 (at 10.240.40.33@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110269/real 1674110269] req@00000000ab61e427 x1755429199291520/t0(0) o400->lustre-OST0001-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110345 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110274/real 1674110274] req@00000000af239d63 x1755429199292224/t0(0) o400->lustre-OST0003-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110350 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110279/real 1674110279] req@0000000035ab7abb x1755429199292864/t0(0) o400->lustre-OST0004-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110355 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110290/real 1674110290] req@000000007368b26b x1755429199293760/t0(0) o400->lustre-OST0000-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110366 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 13 previous similar messages | Link to test (2023-01-19 01:46) |