Editing crashreport #74331

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( dt->do_body_ops->dbo_write ) faileddt_record_writedt_record_write
out_tx_write_exec
out_tx_end
out_handle
tgt_handle_request0
tgt_request_handle
ptlrpc_server_handle_request
ptlrpc_main
kthread
ret_from_fork
4

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0
LustreError: 254996:0:(dt_object.c:395:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: [0x2000284a2:0x1:0x0]
LustreError: 254996:0:(dt_object.c:395:dt_record_write()) LBUG
CPU: 0 PID: 254996 Comm: mdt_out00_003 Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
dt_record_write+0x11c/0x120 [obdclass]
out_tx_write_exec+0xcc/0x410 [ptlrpc]
out_tx_end+0x15f/0x600 [ptlrpc]
out_handle+0x11fe/0x1e30 [ptlrpc]
tgt_handle_request0+0x147/0x770 [ptlrpc]
tgt_request_handle+0x3fd/0xd00 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x2e5/0xd80 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
ptlrpc_main+0x9bf/0xea0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701
Lustre: *** cfs_fail_loc=1701, val=2147483648***
LustreError: 6989:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ff345890b751ba80 x1860806260275840/t352187318279(0) o1000->lustre-MDT0001-mdtlov_UUID@10.240.24.231@tcp:515/0 lens 1312/4320 e 0 to 0 dl 1774608385 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295
Lustre: DEBUG MARKER: sync; sync; sync
Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4071424 flakey 252:0 0 0 1800 1 drop_writes"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 69c9b2e4-1416-41ea-805a-d65ac789797c.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4071424 linear 252:0 0"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): 8 truncates cleaned up
LDISKFS-fs (dm-3): recovery complete
LDISKFS-fs (dm-3): mounted filesystem 69c9b2e4-1416-41ea-805a-d65ac789797c r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm273.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm273.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-157vm273.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-157vm273.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Link to test
replay-single test 80a: DNE: create remote dir, drop update rep from MDT0, fail MDT0
LustreError: 6554:0:(dt_object.c:395:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: [0x200027cd2:0x1:0x0]
LustreError: 6554:0:(dt_object.c:395:dt_record_write()) LBUG
CPU: 0 PID: 6554 Comm: mdt_out00_001 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
dt_record_write+0x11c/0x120 [obdclass]
out_tx_write_exec+0xcc/0x410 [ptlrpc]
out_tx_end+0x15f/0x600 [ptlrpc]
out_handle+0x11fe/0x1e30 [ptlrpc]
tgt_handle_request0+0x147/0x770 [ptlrpc]
tgt_request_handle+0x3fd/0xd00 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x2e5/0xd80 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
ptlrpc_main+0x9bf/0xea0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Lustre: DEBUG MARKER: lctl set_param fail_loc=0x1701
Lustre: *** cfs_fail_loc=1701, val=2147483648***
LustreError: 9351:0:(ldlm_lib.c:3325:target_send_reply_msg()) @@@ dropping reply req@ff4141afb89716c0 x1860806087985536/t347892350983(0) o1000->lustre-MDT0001-mdtlov_UUID@10.240.29.140@tcp:303/0 lens 1312/4320 e 0 to 0 dl 1774608173 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 projid:4294967295
Lustre: DEBUG MARKER: sync; sync; sync
Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4071424 flakey 252:0 0 0 1800 1 drop_writes"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem e1c87796-677f-4791-8e4a-69b1230799ed.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4071424 linear 252:0 0"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): 8 truncates cleaned up
LDISKFS-fs (dm-3): recovery complete
LDISKFS-fs (dm-3): mounted filesystem e1c87796-677f-4791-8e4a-69b1230799ed r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm267.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm267.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm267.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm267.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
debugfs: Directory '10.240.29.138@tcp' with parent 'exports' already present!
debugfs: File 'stats' in directory '/' already present!
debugfs: File 'ldlm_stats' in directory '/' already present!
debugfs: File 'open_files' in directory '/' already present!
Link to test
replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2
LustreError: 5283:0:(dt_object.c:395:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: [0x240007162:0x1:0x0]
LustreError: 5283:0:(dt_object.c:395:dt_record_write()) LBUG
CPU: 0 PID: 5283 Comm: mdt_out00_000 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
dt_record_write+0x11c/0x120 [obdclass]
out_tx_write_exec+0xcc/0x410 [ptlrpc]
out_tx_end+0x162/0x600 [ptlrpc]
out_handle+0x11fe/0x1e30 [ptlrpc]
tgt_handle_request0+0x14a/0x770 [ptlrpc]
tgt_request_handle+0x3fd/0xd00 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x2e8/0xd80 [ptlrpc]
ptlrpc_main+0x9bf/0xea0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xe0/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x2c/0x50
</TASK>
Link to test
replay-single test 117: DNE: cross MDT unlink, fail MDT1 and MDT2
LustreError: 5845:0:(dt_object.c:531:dt_record_write()) ASSERTION( dt->do_body_ops->dbo_write ) failed: [0x240007932:0x1:0x0]
LustreError: 5845:0:(dt_object.c:531:dt_record_write()) LBUG
CPU: 1 PID: 5845 Comm: mdt_out00_000 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
dt_record_write+0x11c/0x120 [obdclass]
out_tx_write_exec+0x90/0x2e0 [ptlrpc]
out_tx_end+0x15f/0x610 [ptlrpc]
out_handle+0x15b9/0x2030 [ptlrpc]
tgt_handle_request0+0x147/0x770 [ptlrpc]
tgt_request_handle+0x3fd/0xd00 [ptlrpc]
ptlrpc_server_handle_request.isra.0+0x2ca/0xda0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
ptlrpc_main+0xa7b/0xfa0 [ptlrpc]
? __pfx_ptlrpc_main+0x10/0x10 [ptlrpc]
kthread+0xdd/0x100
? __pfx_kthread+0x10/0x10
ret_from_fork+0x29/0x50
</TASK>
Link to test
Return to new crashes list