Edit new crash report

Reason	Crashing Function	Where to cut Backtrace	Reports Count
ASSERTION( list_empty(&req->rq_list) ) failed	ptlrpc_send_new_req	ptlrpc_send_new_req ptlrpc_check_set ptlrpcd kthread ret_from_fork	4

Failing Test	Full Crash	Messages before crash	Comment
replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3	LustreError: 4279:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 4279:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 4279 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc] ptlrpc_check_set+0x7eb/0x2080 [ptlrpc] ptlrpcd+0x7a7/0x9c0 [ptlrpc] ? do_wait_intr_irq+0xa0/0xa0 ? ptlrpcd_add_req+0x350/0x350 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40	Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002	Link to test
replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3	LustreError: 4098:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 4098:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 4098 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc] ptlrpc_check_set+0x7eb/0x2080 [ptlrpc] ptlrpcd+0x7a7/0x9c0 [ptlrpc] ? do_wait_intr_irq+0xa0/0xa0 ? ptlrpcd_add_req+0x350/0x350 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40	Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all	Link to test
replay-single test 70f: OSS O_DIRECT recovery with 2 clients	LustreError: 4255:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 4255:0:(client.c:1846:ptlrpc_send_new_req()) LBUG CPU: 0 PID: 4255 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc] ptlrpc_check_set+0x7eb/0x2080 [ptlrpc] ptlrpcd+0x7a7/0x9c0 [ptlrpc] ? do_wait_intr_irq+0xa0/0xa0 ? ptlrpcd_add_req+0x350/0x350 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40	Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: DEBUG MARKER: mount \| grep /mnt/lustre' ' Autotest: Test running for 75 minutes (lustre-reviews_review-dne-zfs-part-6_125445.39) Lustre: DEBUG MARKER: dd bs=1M count=10 if=/dev/urandom of=/tmp/f70f.replay-single Lustre: DEBUG MARKER: md5sum /tmp/f70f.replay-single Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 1 times Lustre: DEBUG MARKER: test_70f failing OST 1 times Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm284.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: trevis-155vm284.trevis.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces..lru_size=clear Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname) Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000 Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285 Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 2 times Lustre: DEBUG MARKER: test_70f failing OST 2 times	Link to test
recovery-mds-scale test failover_ost: failover OST	LustreError: 8006:0:(client.c:1684:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed: LustreError: 8006:0:(client.c:1684:ptlrpc_send_new_req()) LBUG Pid: 8006, comm: ptlrpcd_00_00 4.18.0-372.32.1.el8_6.x86_64 #1 SMP Fri Oct 7 12:35:10 EDT 2022 Call Trace TBD: [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] ptlrpc_send_new_req+0x9d5/0xbc0 [ptlrpc] [<0>] ptlrpc_check_set+0xab8/0x1f60 [ptlrpc] [<0>] ptlrpcd+0x6ca/0xa50 [ptlrpc] [<0>] kthread+0x10a/0x120 [<0>] ret_from_fork+0x35/0x40	Lustre: DEBUG MARKER: PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bi Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: dd on trevis-84vm8 Lustre: DEBUG MARKER: Started client load: dd on trevis-84vm8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: tar on trevis-84vm9 Lustre: DEBUG MARKER: Started client load: tar on trevis-84vm9 Lustre: DEBUG MARKER: cat /tmp/client-load.pid Lustre: DEBUG MARKER: /usr/sbin/lctl mark ==== Check clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1200 Lustre: DEBUG MARKER: ==== Check clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1200 Lustre: DEBUG MARKER: ps auxwww \| grep -v grep \| grep -q run_dd.sh Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait ost3 recovery complete before next failover Lustre: DEBUG MARKER: Wait ost3 recovery complete before next failover Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0000.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0000.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0001.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0001.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0002.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0002.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0003.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0003.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0004.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0004.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0005.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0005.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \.lustre-OST0006.recovery_status 1475 Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete .lustre-OST0006.recovery_status 1475 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Checking clients are in FULL\\|IDLE state before next failover Lustre: DEBUG MARKER: Checking clients are in FULL\|IDLE state before next failover Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm9.trevis.whamcloud.com: executing wait_import_state_mount FULL\\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\.ost_server_uuid,os Lustre: DEBUG MARKER: trevis-84vm9.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0003-osc-[-0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\.ost_server_uuid,os Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\.ost_server_uuid,os Lustre: DEBUG MARKER: trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0003-osc-[-0 Lustre: DEBUG MARKER: trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f].ost_server_uuid,osc.lustre-OST0003-osc-[-0 Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f].ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: /usr/sbin/lctl mark Starting failover on ost3 Lustre: DEBUG MARKER: Starting failover on ost3 LNet: Timeout error while writing to 10.240.40.33:7988. Closing socket: rc = -110 Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1674110310/real 1674110310] req@000000002072c3a5 x1755429199296064/t0(0) o400->lustre-OST0000-osc-ffff904885d98000@10.240.40.33@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110386 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: lustre-OST0000-osc-ffff904885d98000: Connection to lustre-OST0000 (at 10.240.40.33@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110269/real 1674110269] req@00000000ab61e427 x1755429199291520/t0(0) o400->lustre-OST0001-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110345 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110274/real 1674110274] req@00000000af239d63 x1755429199292224/t0(0) o400->lustre-OST0003-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110350 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110279/real 1674110279] req@0000000035ab7abb x1755429199292864/t0(0) o400->lustre-OST0004-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110355 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110290/real 1674110290] req@000000007368b26b x1755429199293760/t0(0) o400->lustre-OST0000-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110366 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0' Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 13 previous similar messages	Link to test

Failing Test

Full Crash

Messages before crash

Comment

replay-single test 112k: DNE: cross MDT rename, fail MDT1,MDT2,MDT3

LustreError: 4279:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed:
LustreError: 4279:0:(client.c:1846:ptlrpc_send_new_req()) LBUG
CPU: 0 PID: 4279 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc]
ptlrpc_check_set+0x7eb/0x2080 [ptlrpc]
ptlrpcd+0x7a7/0x9c0 [ptlrpc]
? do_wait_intr_irq+0xa0/0xa0
? ptlrpcd_add_req+0x350/0x350 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40

Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001
Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002

Link to test

replay-single test 112f: DNE: cross MDT rename, fail MDT1 and MDT3

LustreError: 4098:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed:
LustreError: 4098:0:(client.c:1846:ptlrpc_send_new_req()) LBUG
CPU: 0 PID: 4098 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc]
ptlrpc_check_set+0x7eb/0x2080 [ptlrpc]
ptlrpcd+0x7a7/0x9c0 [ptlrpc]
? do_wait_intr_irq+0xa0/0xa0
? ptlrpcd_add_req+0x350/0x350 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40

Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm53.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all

Link to test

replay-single test 70f: OSS O_DIRECT recovery with 2 clients

LustreError: 4255:0:(client.c:1846:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed:
LustreError: 4255:0:(client.c:1846:ptlrpc_send_new_req()) LBUG
CPU: 0 PID: 4255 Comm: ptlrpcd_rcv Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
ptlrpc_send_new_req+0x9d9/0xbc0 [ptlrpc]
ptlrpc_check_set+0x7eb/0x2080 [ptlrpc]
ptlrpcd+0x7a7/0x9c0 [ptlrpc]
? do_wait_intr_irq+0xa0/0xa0
? ptlrpcd_add_req+0x350/0x350 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40

Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
Autotest: Test running for 75 minutes (lustre-reviews_review-dne-zfs-part-6_125445.39)
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/dev/urandom of=/tmp/f70f.replay-single
Lustre: DEBUG MARKER: md5sum /tmp/f70f.replay-single
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 1 times
Lustre: DEBUG MARKER: test_70f failing OST 1 times
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-155vm284.trevis.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: trevis-155vm284.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: DEBUG MARKER: lctl get_param -n at_max
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: mcreate /mnt/lustre/fsa-$(hostname); rm /mnt/lustre/fsa-$(hostname)
Lustre: DEBUG MARKER: if [ -d /mnt/lustre2 ]; then mcreate /mnt/lustre2/fsa-$(hostname); rm /mnt/lustre2/fsa-$(hostname); fi
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param ldlm.namespaces.*.lru_size=clear
Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-OST0000
Lustre: DEBUG MARKER: md5sum /mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm285
Lustre: DEBUG MARKER: dd oflag=direct bs=1M count=10 if=/tmp/f70f.replay-single of=/mnt/lustre/d70f.replay-single/f70f.replay-single.trevis-155vm284.trevis.whamcloud.com
Lustre: DEBUG MARKER: /usr/sbin/lctl mark test_70f failing OST 2 times
Lustre: DEBUG MARKER: test_70f failing OST 2 times

Link to test

recovery-mds-scale test failover_ost: failover OST

LustreError: 8006:0:(client.c:1684:ptlrpc_send_new_req()) ASSERTION( list_empty(&req->rq_list) ) failed:
LustreError: 8006:0:(client.c:1684:ptlrpc_send_new_req()) LBUG
Pid: 8006, comm: ptlrpcd_00_00 4.18.0-372.32.1.el8_6.x86_64 #1 SMP Fri Oct 7 12:35:10 EDT 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] ptlrpc_send_new_req+0x9d5/0xbc0 [ptlrpc]
[<0>] ptlrpc_check_set+0xab8/0x1f60 [ptlrpc]
[<0>] ptlrpcd+0x6ca/0xa50 [ptlrpc]
[<0>] kthread+0x10a/0x120
[<0>] ret_from_fork+0x35/0x40

Lustre: DEBUG MARKER: PATH=/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/lib64/lustre/utils:/usr/lib64/openmpi/bi
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: dd on trevis-84vm8
Lustre: DEBUG MARKER: Started client load: dd on trevis-84vm8
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Started client load: tar on trevis-84vm9
Lustre: DEBUG MARKER: Started client load: tar on trevis-84vm9
Lustre: DEBUG MARKER: cat /tmp/client-load.pid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark ==== Check clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1200
Lustre: DEBUG MARKER: ==== Check clients loads BEFORE failover -- failure NOT OK ELAPSED=0 DURATION=86400 PERIOD=1200
Lustre: DEBUG MARKER: ps auxwww | grep -v grep | grep -q run_dd.sh
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait ost3 recovery complete before next failover
Lustre: DEBUG MARKER: Wait ost3 recovery complete before next failover
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0000.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0000.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0001.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0001.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0002.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0002.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0003.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0003.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0004.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0004.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0005.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0005.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete \*.lustre-OST0006.recovery_status 1475
Lustre: DEBUG MARKER: trevis-47vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-OST0006.recovery_status 1475
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Checking clients are in FULL\|IDLE state before next failover
Lustre: DEBUG MARKER: Checking clients are in FULL|IDLE state before next failover
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm9.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid,os
Lustre: DEBUG MARKER: trevis-84vm9.trevis.whamcloud.com: executing wait_import_state_mount FULL|IDLE osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0003-osc-[-0
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid,os
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL\|IDLE osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid,os
Lustre: DEBUG MARKER: trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL|IDLE osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0003-osc-[-0
Lustre: DEBUG MARKER: trevis-84vm8.trevis.whamcloud.com: executing wait_import_state_mount FULL|IDLE osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0003-osc-[-0
Lustre: DEBUG MARKER: lctl get_param -n at_max
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0001-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0002-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0003-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0004-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0004-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0005-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0005-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark osc.lustre-OST0006-osc-[-0-9a-f]\*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: osc.lustre-OST0006-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Starting failover on ost3
Lustre: DEBUG MARKER: Starting failover on ost3
LNet: Timeout error while writing to 10.240.40.33:7988. Closing socket: rc = -110
Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1674110310/real 1674110310] req@000000002072c3a5 x1755429199296064/t0(0) o400->lustre-OST0000-osc-ffff904885d98000@10.240.40.33@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110386 ref 1 fl Rpc:eXNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0'
Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: lustre-OST0000-osc-ffff904885d98000: Connection to lustre-OST0000 (at 10.240.40.33@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110269/real 1674110269] req@00000000ab61e427 x1755429199291520/t0(0) o400->lustre-OST0001-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110345 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0'
Lustre: 8006:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110274/real 1674110274] req@00000000af239d63 x1755429199292224/t0(0) o400->lustre-OST0003-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110350 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0'
Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110279/real 1674110279] req@0000000035ab7abb x1755429199292864/t0(0) o400->lustre-OST0004-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110355 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0'
Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 6 previous similar messages
Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1674110290/real 1674110290] req@000000007368b26b x1755429199293760/t0(0) o400->lustre-OST0000-osc-ffff904885d98000@10.240.40.34@tcp:28/4 lens 224/224 e 0 to 1 dl 1674110366 ref 1 fl Rpc:XNQr/0/ffffffff rc 0/-1 job:'kworker/u4:1.0'
Lustre: 8007:0:(client.c:2305:ptlrpc_expire_one_request()) Skipped 13 previous similar messages

Link to test

Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below):
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below):
Limit to a test: (Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Editing crashreport #67941

Added fields:

Failures list (last 100):