Editing crashreport #73277

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( nm->nm_md_stats ) failedmdt_counter_incrmdt_counter_incr
mdt_statfs
tgt_request_handle
ptlrpc_server_handle_request
ptlrpc_main
kthread
ret_from_fork
3

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanity-sec test 15: test id mapping
LustreError: 245593:0:(mdt_lproc.c:1610:mdt_counter_incr()) ASSERTION( nm->nm_md_stats ) failed:
LustreError: 245593:0:(mdt_lproc.c:1610:mdt_counter_incr()) LBUG
CPU: 1 PID: 245593 Comm: mdt_out00_001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
mdt_counter_incr+0x188/0x190 [mdt]
mdt_statfs+0x4b3/0x8c0 [mdt]
? tgt_request_preprocess.isra.30+0x21e/0x8c0 [ptlrpc]
tgt_request_handle+0x3f4/0x1b80 [ptlrpc]
ptlrpc_server_handle_request+0x27b/0xcd0 [ptlrpc]
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xc81/0x1560 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x5b0/0x5b0 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.squash_uid
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.squash_gid
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.51592_0.id
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.51592_1.id
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.51592_2.id
Link to test
sanity-sec test 15: test id mapping
LustreError: 214944:0:(mdt_lproc.c:1610:mdt_counter_incr()) ASSERTION( nm->nm_md_stats ) failed:
LustreError: 214944:0:(mdt_lproc.c:1610:mdt_counter_incr()) LBUG
CPU: 0 PID: 214944 Comm: mdt_out00_002 Kdump: loaded Tainted: P OE -------- - - 4.18.0-553.50.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
mdt_counter_incr+0x188/0x190 [mdt]
mdt_statfs+0x4b3/0x8c0 [mdt]
? tgt_request_preprocess.isra.30+0x21e/0x8c0 [ptlrpc]
tgt_request_handle+0x3f4/0x1b80 [ptlrpc]
ptlrpc_server_handle_request+0x27b/0xcd0 [ptlrpc]
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xc81/0x1560 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x5b0/0x5b0 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.squash_uid
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.squash_gid
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.59140_0.id
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.59140_1.id
Autotest: Test running for 105 minutes (lustre-reviews_review-dne-zfs-part-2_113101.13)
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.59140_2.id
Link to test
sanity-sec test 15: test id mapping
LustreError: 12247:0:(mdt_lproc.c:1610:mdt_counter_incr()) ASSERTION( nm->nm_md_stats ) failed:
LustreError: 12247:0:(mdt_lproc.c:1610:mdt_counter_incr()) LBUG
CPU: 1 PID: 12247 Comm: mdt_out00_001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
mdt_counter_incr+0x188/0x190 [mdt]
mdt_statfs+0x4b3/0x8c0 [mdt]
? tgt_request_preprocess.isra.30+0x21e/0x8c0 [ptlrpc]
tgt_request_handle+0x3f4/0x1b80 [ptlrpc]
ptlrpc_server_handle_request+0x27b/0xcd0 [ptlrpc]
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xc81/0x1560 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x5b0/0x5b0 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.squash_uid
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.squash_gid
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.41316_0.id
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.41316_1.id
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.41316_2.id
LNet: Host 10.240.39.186 reset our connection while we were sending data; it may have rebooted: rc = -104
Lustre: 10968:0:(client.c:2451:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1746449218/real 1746449218] req@ffff8e3f82b92d80 x1831277336825600/t0(0) o400->lustre-OST0000-osc-MDT0003@10.240.39.186@tcp:28/4 lens 224/224 e 0 to 1 dl 1746449234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
Lustre: 10968:0:(client.c:2451:ptlrpc_expire_one_request()) Skipped 12 previous similar messages
Lustre: lustre-OST0000-osc-MDT0003: Connection to lustre-OST0000 (at 10.240.39.186@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 16 previous similar messages
Autotest: Killing test framework, node(s) in the cluster crashed (lustre-reviews_review-dne-selinux-ssk-part-2_113101.40)
Autotest: Sleeping to ensure other nodes in the cluster have not crashed (lustre-reviews_review-dne-selinux-ssk-part-2_113101.40)
Lustre: lustre-MDT0003: haven't heard from client lustre-MDT0003-lwp-OST0001_UUID (at 10.240.39.186@tcp) in 101 seconds. I think it's dead, and I am evicting it. exp ffff8e3f59284000, cur 1746449309 deadline 1746449308 last 1746449208
Lustre: Skipped 1 previous similar message
Link to test
Return to new crashes list