Editing crashreport #74268

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( lod->lod_initialized ) failedlod_sub_init_llogslod_prepare
mdd_prepare
mdt_prepare
server_start_targets
server_fill_super
lustre_fill_super
mount_nodev
legacy_get_tree
vfs_get_tree
do_new_mount
__x64_sys_mount
do_syscall_64
entry_SYSCALL_64_after_hwframe
2

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanity-sec test 27a: test fileset in various nodemaps
LustreError: 327380:0:(lod_dev.c:1220:lod_sub_init_llogs()) ASSERTION( lod->lod_initialized ) failed:
LustreError: 327380:0:(lod_dev.c:1220:lod_sub_init_llogs()) LBUG
CPU: 0 PID: 327380 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lod_prepare+0x7d8/0x7e0 [lod]
mdd_prepare+0x110/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa91/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_niduuid_destroy+0x7c/0x100 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0x7fe/0x17c0 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? arch_exit_to_user_mode_prepare.constprop.0+0x6c/0xa0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fdcb850f60e
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.default.admin_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin=1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin=1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name default --fileset /thisisaverylongsubdirtotestlongfilesetsandtotestmultiplefilesetfragmentsonthenodemapiam_default
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.fileset | awk '/primary/ { if ($3 == "/thisisaverylongsubdirtotestlongfilesetsandtotestmultiplefilesetfragmentsonthenodemapiam_default") print $3 }'
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.fileset
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
Lustre: lustre-MDT0000: Not available for connect from 10.240.39.126@tcp (stopping)
Lustre: Skipped 6 previous similar messages
Lustre: lustre-MDT0000: Not available for connect from 10.240.39.126@tcp (stopping)
Lustre: Skipped 7 previous similar messages
LDISKFS-fs (dm-2): unmounting filesystem 94006fad-3b32-4a1f-9ade-1e294a32c01b.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm15.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm15.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
LNet: 310813:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNetError: 310813:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
LNet: Removed LNI 10.240.39.127@tcp
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: trevis-154vm15.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn48_17_gd6b0c1a
LNet: Added LNI 10.240.39.127@tcp [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-2): mounted filesystem 94006fad-3b32-4a1f-9ade-1e294a32c01b r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P raft_num_voting_nodes=3
Lustre: 314034:0:(mgs_llog.c:1350:mgs_modify_param()) MGS: modify general/raft_num_voting_nodes=3 (mode = 0) failed: rc = -17
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Autotest: Test running for 90 minutes (lustre-b_es-reviews_review-exa7-part-1_27666.151)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name default --fileset clear
Lustre: 315661:0:(mgs_llog.c:1350:mgs_modify_param()) MGS: modify general/nodemap.default.fileset= (mode = 1) failed: rc = -2
Lustre: 315661:0:(mgs_llog.c:1350:mgs_modify_param()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.fileset | awk '/primary/ { if ($3 == "/thisisaverylongsubdirtotestlongfilesetsandtotestmultiplefilesetfragmentsonthenodemapiam_default") print $3 }'
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.default.admin_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.default.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm15.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm15.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param mdt.*.identity_upcall=NONE
Lustre: 318507:0:(mdt_lproc.c:331:identity_upcall_store()) lustre-MDT0000: disable "identity_upcall" with ACL enabled maybe cause unexpected "EACCESS"
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c0 --range 10.240.39.124@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60003:60000
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60003:60000
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype uid --idmap 60004:60002
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype gid --idmap 60004:60002
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add c1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_range --name c1 --range 10.240.39.125@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60005:60001
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60005:60001
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype uid --idmap 60006:60002
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c1 --idtype gid --idmap 60006:60002
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name default --property trusted --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_activate 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.admin_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.c0.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin=1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin=0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted=0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.trusted_nodemap
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_set_fileset --name c0 --fileset /thisisaverylongsubdirtotestlongfilesetsandtotestmultiplefilesetfragmentsonthenodemapiam_c0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset | awk '/primary/ { if ($3 == "/thisisaverylongsubdirtotestlongfilesetsandtotestmultiplefilesetfragmentsonthenodemapiam_c0") print $3 }'
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.fileset
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property admin --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property trusted --value 0
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property map_mode --value projid
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_add_idmap --name c0 --idtype projid --idmap 1:1
Lustre: DEBUG MARKER: /usr/sbin/lctl nodemap_modify --name c0 --property deny_unknown --value 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n nodemap.active
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param nodemap.c0.deny_unknown
Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids | grep -w tcp | cut -f 1 -d @
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1
LDISKFS-fs (dm-2): unmounting filesystem 94006fad-3b32-4a1f-9ade-1e294a32c01b.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm15.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm15.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing unload_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing unload_modules_local
Key type lgssc unregistered
LNet: 325269:0:(lib-ptl.c:989:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNetError: 325269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found
LNet: Removed LNI 10.240.39.127@tcp
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: trevis-154vm15.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm16.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-154vm17.trevis.whamcloud.com: executing load_modules_local
Lustre: DEBUG MARKER: trevis-154vm16.trevis.whamcloud.com: executing load_modules_local
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.16.0_ddn48_17_gd6b0c1a
LNet: Added LNI 10.240.39.127@tcp [8/256/0/180]
Key type lgssc registered
Lustre: Echo OBD driver; http://www.lustre.org/
alg: No test for zstdfast (zstdfast-lustre-generic)
alg: No test for zstdfast (zstdfast-lustre-scomp)
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-2): mounted filesystem 94006fad-3b32-4a1f-9ade-1e294a32c01b r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Link to test
replay-single test 53b: |X| open request while two MDC requests in flight
LustreError: 366852:0:(lod_dev.c:1220:lod_sub_init_llogs()) ASSERTION( lod->lod_initialized ) failed:
LustreError: 366852:0:(lod_dev.c:1220:lod_sub_init_llogs()) LBUG
CPU: 0 PID: 366852 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lod_prepare+0x7d8/0x7e0 [lod]
mdd_prepare+0x110/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x17d/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xac6/0xc00 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x1da/0x370
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_niduuid_destroy+0x7c/0x100 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0x7fe/0x17c0 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xc0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x59/0x90
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7fdfbb70f03e
Lustre: DEBUG MARKER: lctl set_param fail_loc=0x80000107
Lustre: *** cfs_fail_loc=107, val=2147483648***
Lustre: DEBUG MARKER: lctl set_param fail_loc=0
Lustre: DEBUG MARKER: sync; sync; sync
Lustre: DEBUG MARKER: /usr/sbin/lctl --device lustre-MDT0000 notransno
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 flakey 252:0 0 0 1800 1 drop_writes"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: /usr/sbin/lctl mark mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-2): unmounting filesystem 17e5cba6-dc2c-4bed-9f75-c1d93b78551a.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: dmsetup table /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup suspend --nolockfs --noflush /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: dmsetup load /dev/mapper/mds1_flakey --table "0 4194304 linear 252:0 0"
Lustre: DEBUG MARKER: dmsetup resume /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-2): recovery complete
LDISKFS-fs (dm-2): mounted filesystem 17e5cba6-dc2c-4bed-9f75-c1d93b78551a r/w with ordered data mode. Quota mode: journalled.
Link to test
Return to new crashes list