LustreError: 236047:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 236047:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 236047 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? mntput_no_expire+0x4a/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fe7e970f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 233921:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 233921:0:(obd_class.h:478:obd_check_dev()) Skipped 245 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem a5297573-b84f-478b-9782-dfadcc2cb8d8.
Lustre: server umount lustre-MDT0000 complete
LustreError: 209618:0:(ldlm_lib.c:1242:target_handle_connect()) lustre-MDT0000: not available for connect from 10.240.47.118@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: 209618:0:(ldlm_lib.c:1242:target_handle_connect()) Skipped 132 previous similar messages
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem a5297573-b84f-478b-9782-dfadcc2cb8d8 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem a5297573-b84f-478b-9782-dfadcc2cb8d8.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 3055f7f2-8755-4dd8-a881-b46368c26d60 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 3055f7f2-8755-4dd8-a881-b46368c26d60.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 3055f7f2-8755-4dd8-a881-b46368c26d60 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 3055f7f2-8755-4dd8-a881-b46368c26d60.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 3055f7f2-8755-4dd8-a881-b46368c26d60 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 220493:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220493:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 220493 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? fpregs_restore_userregs+0x47/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f2d0450f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 218484:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 218484:0:(obd_class.h:478:obd_check_dev()) Skipped 245 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem c619c87a-00d7-4006-9ed7-3baa8133e217.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem c619c87a-00d7-4006-9ed7-3baa8133e217 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem c619c87a-00d7-4006-9ed7-3baa8133e217.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Autotest: Test running for 70 minutes (lustre-reviews_review-dne-part-2_122950.43)
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 9 previous similar messages
LDISKFS-fs (dm-3): mounted filesystem ac133eaf-e520-4f75-9e59-9687f91f798c r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem ac133eaf-e520-4f75-9e59-9687f91f798c.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem ac133eaf-e520-4f75-9e59-9687f91f798c r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem ac133eaf-e520-4f75-9e59-9687f91f798c.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem ac133eaf-e520-4f75-9e59-9687f91f798c r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 184074:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 184074:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 184074 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? __memcg_slab_free_hook+0xea/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_free+0x3f1/0x420
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xc0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f886390f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 7a18d53a-5c11-4a34-9650-9f6b2da476fa.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 2c37a4be-f784-4fa0-89b1-d3ff97d6fc9e.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 7a18d53a-5c11-4a34-9650-9f6b2da476fa r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 7a18d53a-5c11-4a34-9650-9f6b2da476fa.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 2c37a4be-f784-4fa0-89b1-d3ff97d6fc9e r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 2c37a4be-f784-4fa0-89b1-d3ff97d6fc9e.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 7a18d53a-5c11-4a34-9650-9f6b2da476fa r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 172274:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 172274:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 172274 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? __pfx_file_free_rcu+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0x117/0x2b0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f4c8270f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 170502:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 170502:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem a75d04c9-ac61-42ab-a995-5a3d5a29c80a.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.28.100@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 387c6e5e-28a5-45d6-9cf1-77c898d62963.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Autotest: Test running for 95 minutes (lustre-reviews_review-dne-part-7_122950.48)
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem a75d04c9-ac61-42ab-a995-5a3d5a29c80a r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem a75d04c9-ac61-42ab-a995-5a3d5a29c80a.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 387c6e5e-28a5-45d6-9cf1-77c898d62963 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 387c6e5e-28a5-45d6-9cf1-77c898d62963.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem a75d04c9-ac61-42ab-a995-5a3d5a29c80a r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 220551:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220551:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 220551 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0x117/0x2b0
? srso_alias_return_thunk+0x5/0xfbef5
? wait_task_zombie+0x14a/0x580
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? __handle_mm_fault+0x2fb/0x690
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f505f70f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem f8425616-2787-4a1b-bee5-faebfc814948.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem f8425616-2787-4a1b-bee5-faebfc814948 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem f8425616-2787-4a1b-bee5-faebfc814948.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 49af12f2-ad5f-46a1-8d97-34519569b221 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 49af12f2-ad5f-46a1-8d97-34519569b221.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 49af12f2-ad5f-46a1-8d97-34519569b221 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 49af12f2-ad5f-46a1-8d97-34519569b221.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 49af12f2-ad5f-46a1-8d97-34519569b221 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 172096:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 172096:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 172096 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __unfreeze_partials+0x173/0x1e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fe2c670f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 170365:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 170365:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem a05f6700-3b0d-4986-ba56-d92a9629f5fe.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 8 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.28.1@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem e472799f-1ded-42be-980b-4c6abaf34519.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem a05f6700-3b0d-4986-ba56-d92a9629f5fe r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem a05f6700-3b0d-4986-ba56-d92a9629f5fe.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem e472799f-1ded-42be-980b-4c6abaf34519 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem e472799f-1ded-42be-980b-4c6abaf34519.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem a05f6700-3b0d-4986-ba56-d92a9629f5fe r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 236275:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 236275:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 236275 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __slab_free+0xcb/0x310
? mgc_set_info_async+0x5fd/0x670 [mgc]
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? finish_task_switch.isra.0+0x8c/0x2a0
? srso_alias_return_thunk+0x5/0xfbef5
? __schedule+0x231/0x4a0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? prepare_to_wait_event+0x5d/0x180
? srso_alias_return_thunk+0x5/0xfbef5
? finish_wait+0x41/0x80
? srso_alias_return_thunk+0x5/0xfbef5
? mutex_lock+0xe/0x30
? srso_alias_return_thunk+0x5/0xfbef5
? pipe_read+0x2b1/0x490
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_autoremove_wake_function+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? rseq_get_rseq_cs+0x1d/0x240
? srso_alias_return_thunk+0x5/0xfbef5
? rseq_ip_fixup+0x6e/0x1a0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xc0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f7abad0f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 234149:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 234149:0:(obd_class.h:478:obd_check_dev()) Skipped 245 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 8d07af62-ea7a-4bae-b695-ddb2541209e9.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 8d07af62-ea7a-4bae-b695-ddb2541209e9 r/w with ordered data mode. Quota mode: journalled.
LustreError: 232883:0:(ldlm_lib.c:1242:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: 232883:0:(ldlm_lib.c:1242:target_handle_connect()) Skipped 176 previous similar messages
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 8d07af62-ea7a-4bae-b695-ddb2541209e9.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem faf6d98a-2103-4abf-8b3d-0ceb379bc22d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem faf6d98a-2103-4abf-8b3d-0ceb379bc22d.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem faf6d98a-2103-4abf-8b3d-0ceb379bc22d r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem faf6d98a-2103-4abf-8b3d-0ceb379bc22d.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem faf6d98a-2103-4abf-8b3d-0ceb379bc22d r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings
LustreError: MGC10.240.47.193@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 9 previous similar messages

LustreError: 220452:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220452:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 220452 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7eff0f70f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem faa9e78c-fc1f-49fe-9403-914a6c578ab5.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem faa9e78c-fc1f-49fe-9403-914a6c578ab5 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem faa9e78c-fc1f-49fe-9403-914a6c578ab5.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
LustreError: 195745:0:(ldlm_lib.c:1242:target_handle_connect()) lustre-MDT0000: not available for connect from 10.240.46.240@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: 195745:0:(ldlm_lib.c:1242:target_handle_connect()) Skipped 199 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem b23584a6-225b-4d5f-9f34-4ece24ca8132 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem b23584a6-225b-4d5f-9f34-4ece24ca8132.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem b23584a6-225b-4d5f-9f34-4ece24ca8132 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem b23584a6-225b-4d5f-9f34-4ece24ca8132.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem b23584a6-225b-4d5f-9f34-4ece24ca8132 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 184180:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 184180:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 184180 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __unfreeze_partials+0x1a2/0x1f0
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fdb7430f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 5f706b70-0825-4265-9434-042cceaa438f.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.23.3@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 932efde1-be01-42c3-abdd-c59b56dd9e60.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 5f706b70-0825-4265-9434-042cceaa438f r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 5f706b70-0825-4265-9434-042cceaa438f.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 932efde1-be01-42c3-abdd-c59b56dd9e60 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 932efde1-be01-42c3-abdd-c59b56dd9e60.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 5f706b70-0825-4265-9434-042cceaa438f r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 172003:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 172003:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 172003 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? fpregs_restore_userregs+0x47/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? prepare_to_wait_event+0x5d/0x180
? srso_alias_return_thunk+0x5/0xfbef5
? finish_wait+0x41/0x80
? srso_alias_return_thunk+0x5/0xfbef5
? mutex_lock+0xe/0x30
? srso_alias_return_thunk+0x5/0xfbef5
? pipe_read+0x1c7/0x4d0
? __pfx_autoremove_wake_function+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? rseq_get_rseq_cs+0x1d/0x240
? srso_alias_return_thunk+0x5/0xfbef5
? rseq_ip_fixup+0x6e/0x1a0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f9f7a10f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 170271:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 170271:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem c803fa58-12d0-41c9-963b-e155490c6d50.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 9 previous similar messages
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.22.231@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem a90ac2bf-79c1-4c31-aa1b-bed167672867.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem c803fa58-12d0-41c9-963b-e155490c6d50 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem c803fa58-12d0-41c9-963b-e155490c6d50.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem a90ac2bf-79c1-4c31-aa1b-bed167672867 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem a90ac2bf-79c1-4c31-aa1b-bed167672867.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem c803fa58-12d0-41c9-963b-e155490c6d50 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 235995:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 235995:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 235995 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? fsnotify_grab_connector+0x49/0x80
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? wait_task_zombie+0x14a/0x580
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xc0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f1dce70f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem c2236c15-b773-465d-b2f1-1860d238aa36.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem c2236c15-b773-465d-b2f1-1860d238aa36 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem c2236c15-b773-465d-b2f1-1860d238aa36.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 2dfd3056-f6d7-41ba-98b2-eb5132e8721d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 2dfd3056-f6d7-41ba-98b2-eb5132e8721d.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 2dfd3056-f6d7-41ba-98b2-eb5132e8721d r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 2dfd3056-f6d7-41ba-98b2-eb5132e8721d.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 2dfd3056-f6d7-41ba-98b2-eb5132e8721d r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 220567:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220567:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 220567 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __dentry_kill+0x12b/0x170
? srso_alias_return_thunk+0x5/0xfbef5
? rcu_nocb_try_bypass+0x5e/0x460
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_file_free_rcu+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0x117/0x2b0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fe24c30f13e

Lustre: 195860:0:(osd_internal.h:1368:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 320, rollback = 2
Lustre: 195860:0:(osd_internal.h:1368:osd_trans_exec_op()) Skipped 1130 previous similar messages
Lustre: 195860:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0
Lustre: 195860:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1130 previous similar messages
Lustre: 195860:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 5/5/1, xattr_set: 7/320/0
Lustre: 195860:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1130 previous similar messages
Lustre: 195860:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 6/38/0, punch: 0/0/0, quota 1/3/0
Lustre: 195860:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1130 previous similar messages
Lustre: 195860:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/1
Lustre: 195860:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1130 previous similar messages
Lustre: 195860:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 195860:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1130 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 75d87ba6-36b5-4e61-9d91-10ee3a329b20.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 75d87ba6-36b5-4e61-9d91-10ee3a329b20 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 75d87ba6-36b5-4e61-9d91-10ee3a329b20.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 8019ce29-bab8-45cb-a1ca-4988f632ed5d r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 8019ce29-bab8-45cb-a1ca-4988f632ed5d.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 8019ce29-bab8-45cb-a1ca-4988f632ed5d r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 8019ce29-bab8-45cb-a1ca-4988f632ed5d.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 8019ce29-bab8-45cb-a1ca-4988f632ed5d r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 183902:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 183902:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 183902 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? fsnotify_grab_connector+0x49/0x80
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? wait_task_zombie+0x14a/0x580
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fabc770f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 181989:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 181989:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem daa6663a-2143-4928-8bb8-3c6dd299367c.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 9 previous similar messages
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.23.132@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem f6d8952c-522b-4eb9-8098-833b714dd07e.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Autotest: Test running for 85 minutes (lustre-reviews_review-dne-part-7_122657.57)
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem daa6663a-2143-4928-8bb8-3c6dd299367c r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem daa6663a-2143-4928-8bb8-3c6dd299367c.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem f6d8952c-522b-4eb9-8098-833b714dd07e r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem f6d8952c-522b-4eb9-8098-833b714dd07e.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem daa6663a-2143-4928-8bb8-3c6dd299367c r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 171909:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 171909:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 171909 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? mod_delayed_work_on+0x5e/0x90
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __unfreeze_partials+0x173/0x1e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? __pfx_delayed_put_task_struct+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0x117/0x2b0
? srso_alias_return_thunk+0x5/0xfbef5
? wait_task_zombie+0x14a/0x580
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f121a30f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 170178:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 170178:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 18caff4b-5d9c-4435-b41f-93a5f6d792c7.
Lustre: server umount lustre-MDT0000 complete
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 8 previous similar messages
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.42.74@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem eea48547-26c9-4d6f-9879-842c58c82d24.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 18caff4b-5d9c-4435-b41f-93a5f6d792c7 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 18caff4b-5d9c-4435-b41f-93a5f6d792c7.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem eea48547-26c9-4d6f-9879-842c58c82d24 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem eea48547-26c9-4d6f-9879-842c58c82d24.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 18caff4b-5d9c-4435-b41f-93a5f6d792c7 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 236127:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 236127:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 236127 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __alloc_pages+0xf2/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? __mod_memcg_lruvec_state+0x8a/0x120
? srso_alias_return_thunk+0x5/0xfbef5
? __mod_lruvec_page_state+0x97/0x150
? srso_alias_return_thunk+0x5/0xfbef5
? folio_add_new_anon_rmap+0x41/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_anonymous_page+0x1bb/0x3e0
? srso_alias_return_thunk+0x5/0xfbef5
? __handle_mm_fault+0x2fe/0x650
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f0afb10f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 1fd2d2cb-63fe-4b02-973b-00d4e7947231.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 1fd2d2cb-63fe-4b02-973b-00d4e7947231 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 1fd2d2cb-63fe-4b02-973b-00d4e7947231.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 8 previous similar messages
LDISKFS-fs (dm-3): mounted filesystem ac008bab-af1c-4fc8-8947-909530bffbf7 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem ac008bab-af1c-4fc8-8947-909530bffbf7.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem ac008bab-af1c-4fc8-8947-909530bffbf7 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem ac008bab-af1c-4fc8-8947-909530bffbf7.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem ac008bab-af1c-4fc8-8947-909530bffbf7 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 220049:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220049:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 220049 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f279a90f13e

Lustre: 195383:0:(osd_internal.h:1368:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 320, rollback = 2
Lustre: 195383:0:(osd_internal.h:1368:osd_trans_exec_op()) Skipped 1127 previous similar messages
Lustre: 195383:0:(osd_handler.c:2082:osd_trans_dump_creds()) create: 1/4/0, destroy: 1/4/0
Lustre: 195383:0:(osd_handler.c:2082:osd_trans_dump_creds()) Skipped 1127 previous similar messages
Lustre: 195383:0:(osd_handler.c:2089:osd_trans_dump_creds()) attr_set: 5/5/0, xattr_set: 7/320/0
Lustre: 195383:0:(osd_handler.c:2089:osd_trans_dump_creds()) Skipped 1127 previous similar messages
Lustre: 195383:0:(osd_handler.c:2096:osd_trans_dump_creds()) write: 6/38/0, punch: 0/0/0, quota 1/3/0
Lustre: 195383:0:(osd_handler.c:2096:osd_trans_dump_creds()) Skipped 1127 previous similar messages
Lustre: 195383:0:(osd_handler.c:2106:osd_trans_dump_creds()) insert: 2/33/0, delete: 2/5/1
Lustre: 195383:0:(osd_handler.c:2106:osd_trans_dump_creds()) Skipped 1127 previous similar messages
Lustre: 195383:0:(osd_handler.c:2113:osd_trans_dump_creds()) ref_add: 1/1/0, ref_del: 2/2/1
Lustre: 195383:0:(osd_handler.c:2113:osd_trans_dump_creds()) Skipped 1127 previous similar messages
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 16d6fa80-a74c-4f37-9e39-94fb267418b1.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 16d6fa80-a74c-4f37-9e39-94fb267418b1 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 16d6fa80-a74c-4f37-9e39-94fb267418b1.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem dfcfabf1-f186-4d60-8df1-f562b955af63 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem dfcfabf1-f186-4d60-8df1-f562b955af63.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem dfcfabf1-f186-4d60-8df1-f562b955af63 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem dfcfabf1-f186-4d60-8df1-f562b955af63.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem dfcfabf1-f186-4d60-8df1-f562b955af63 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 236330:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 236330:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 236330 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __slab_free+0xcb/0x310
? mgc_set_info_async+0x5fd/0x670 [mgc]
? lustre_start_mgc+0xa4d/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? fpregs_restore_userregs+0x47/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f596150f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 234203:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 234203:0:(obd_class.h:478:obd_check_dev()) Skipped 245 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 8f06c7c5-2c8f-4b15-acab-5469815b3626.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 8f06c7c5-2c8f-4b15-acab-5469815b3626 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 8f06c7c5-2c8f-4b15-acab-5469815b3626.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem f6e1963b-49ef-44aa-bd54-a80dbb914b21 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem f6e1963b-49ef-44aa-bd54-a80dbb914b21.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem f6e1963b-49ef-44aa-bd54-a80dbb914b21 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem f6e1963b-49ef-44aa-bd54-a80dbb914b21.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem f6e1963b-49ef-44aa-bd54-a80dbb914b21 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 220127:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220127:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 220127 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f9952f0f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem fcbaac8f-5ce1-438f-8cad-03d71322f4a9.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem fcbaac8f-5ce1-438f-8cad-03d71322f4a9 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem fcbaac8f-5ce1-438f-8cad-03d71322f4a9.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem cf795b26-17ec-409a-9067-0d7b9b5f667e r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem cf795b26-17ec-409a-9067-0d7b9b5f667e.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem cf795b26-17ec-409a-9067-0d7b9b5f667e r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem cf795b26-17ec-409a-9067-0d7b9b5f667e.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem cf795b26-17ec-409a-9067-0d7b9b5f667e r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 172012:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 172012:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 172012 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x50/0x3c0 [mdt]
server_start_targets+0xa8f/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? kick_pool+0x65/0x140
? _raw_spin_unlock+0xa/0x30
? __queue_work+0x111/0x390
? mod_delayed_work_on+0x57/0x90
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? obd_connect.constprop.0+0x90/0x340 [obdclass]
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x44/0x90
legacy_get_tree+0x27/0x50
vfs_get_tree+0x25/0xd0
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5f/0xf0
? mm_account_fault+0x6c/0x100
? handle_mm_fault+0x116/0x270
? do_user_addr_fault+0x1d6/0x6a0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fee3cd0f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 2a042c5a-10a3-48d5-9949-a79dfd414cdd.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 827c6a4c-a7ad-44b6-8df4-789f45450cc0.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 2a042c5a-10a3-48d5-9949-a79dfd414cdd r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 2a042c5a-10a3-48d5-9949-a79dfd414cdd.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 827c6a4c-a7ad-44b6-8df4-789f45450cc0 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 827c6a4c-a7ad-44b6-8df4-789f45450cc0.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 2a042c5a-10a3-48d5-9949-a79dfd414cdd r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 184293:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 184293:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 184293 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __unfreeze_partials+0x1a2/0x1f0
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? __handle_mm_fault+0x2fe/0x650
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fe66b90f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Autotest: Test running for 85 minutes (lustre-reviews_review-dne-part-7_122633.57)
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 182424:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 182424:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 52d864b8-58e6-4c89-968d-c4ba12e08649.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.23.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 68618a5f-c0e5-49c9-ac16-51d3e1db8002.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 52d864b8-58e6-4c89-968d-c4ba12e08649 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 52d864b8-58e6-4c89-968d-c4ba12e08649.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 68618a5f-c0e5-49c9-ac16-51d3e1db8002 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 68618a5f-c0e5-49c9-ac16-51d3e1db8002.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 52d864b8-58e6-4c89-968d-c4ba12e08649 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 184002:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 184002:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 184002 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? __slab_free+0xcb/0x310
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? __handle_mm_fault+0x2fe/0x650
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f89ee50f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 7 previous similar messages
LustreError: 182134:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 182134:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 7e123498-81e1-4534-b5e9-970bc8d4370c.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.31.156@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 58f53bc7-298e-49e7-99bc-ddad1519bc34.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 7e123498-81e1-4534-b5e9-970bc8d4370c r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 7e123498-81e1-4534-b5e9-970bc8d4370c.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 58f53bc7-298e-49e7-99bc-ddad1519bc34 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 58f53bc7-298e-49e7-99bc-ddad1519bc34.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 7e123498-81e1-4534-b5e9-970bc8d4370c r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 172281:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 172281:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 172281 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? wait_task_zombie+0x14a/0x580
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? fpregs_restore_userregs+0x47/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f465f50f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 170548:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 170548:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 107d90e9-5fea-4d46-bda8-c51f8240ec80.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.31.151@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 1c70c93b-a06b-4a3f-b93d-56a2fdb5145e.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 107d90e9-5fea-4d46-bda8-c51f8240ec80 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 107d90e9-5fea-4d46-bda8-c51f8240ec80.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 1c70c93b-a06b-4a3f-b93d-56a2fdb5145e r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 1c70c93b-a06b-4a3f-b93d-56a2fdb5145e.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 107d90e9-5fea-4d46-bda8-c51f8240ec80 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 220494:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 220494:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 220494 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? __pfx_mdt_prepare+0x10/0x10 [mdt]
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? rseq_get_rseq_cs+0x1d/0x240
? srso_alias_return_thunk+0x5/0xfbef5
? rseq_ip_fixup+0x6e/0x1a0
? srso_alias_return_thunk+0x5/0xfbef5
? __rseq_handle_notify_resume+0x26/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? exit_to_user_mode_loop+0xd9/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fc027b0f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 98384c01-0832-4668-8b30-0bc868da4a3a.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 98384c01-0832-4668-8b30-0bc868da4a3a r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 98384c01-0832-4668-8b30-0bc868da4a3a.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 9 previous similar messages
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 8d8990da-adb3-48f5-8398-c6bd19ccf2b8 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 8d8990da-adb3-48f5-8398-c6bd19ccf2b8.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 8d8990da-adb3-48f5-8398-c6bd19ccf2b8 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 8d8990da-adb3-48f5-8398-c6bd19ccf2b8.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 8d8990da-adb3-48f5-8398-c6bd19ccf2b8 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 235877:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 235877:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 0 PID: 235877 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __slab_free+0xcb/0x310
? mgc_set_info_async+0x5fd/0x670 [mgc]
? lustre_start_mgc+0xa4d/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? exit_to_user_mode_prepare+0xef/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f6552b0f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 5ff7fcef-9196-4a31-b673-d5b63f13fda1.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 5ff7fcef-9196-4a31-b673-d5b63f13fda1 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 5ff7fcef-9196-4a31-b673-d5b63f13fda1.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem a183afcd-cda9-4368-980f-052d506e3904 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem a183afcd-cda9-4368-980f-052d506e3904.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem a183afcd-cda9-4368-980f-052d506e3904 r/w with ordered data mode. Quota mode: journalled.
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 7 previous similar messages
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem a183afcd-cda9-4368-980f-052d506e3904.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem a183afcd-cda9-4368-980f-052d506e3904 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 183994:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 183994:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 183994 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
? kmem_cache_alloc+0x16d/0x310
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __unfreeze_partials+0x1a2/0x1f0
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7ffaf190f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 182124:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 182124:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem f1c6799d-c068-4b37-86bc-9ea85960147a.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.23.78@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 22f25ef5-8694-46b4-a5a7-4c22fb1d4a19.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem f1c6799d-c068-4b37-86bc-9ea85960147a r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem f1c6799d-c068-4b37-86bc-9ea85960147a.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 22f25ef5-8694-46b4-a5a7-4c22fb1d4a19 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 22f25ef5-8694-46b4-a5a7-4c22fb1d4a19.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem f1c6799d-c068-4b37-86bc-9ea85960147a r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 172088:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 172088:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 172088 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __unfreeze_partials+0x173/0x1e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? folio_add_new_anon_rmap+0x44/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_anonymous_page+0x25a/0x410
? srso_alias_return_thunk+0x5/0xfbef5
? __handle_mm_fault+0x2fb/0x690
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x116/0x270
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x1d6/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fa88c30f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Autotest: Test running for 85 minutes (lustre-reviews_review-dne-part-7_122582.49)
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 170357:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 170357:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 90d12a87-838b-4e4f-8475-fded6d983294.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.46.136@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 2460ba2e-adf0-474d-aed0-763aa171b9de.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 90d12a87-838b-4e4f-8475-fded6d983294 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 90d12a87-838b-4e4f-8475-fded6d983294.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 2460ba2e-adf0-474d-aed0-763aa171b9de r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 2460ba2e-adf0-474d-aed0-763aa171b9de.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 90d12a87-838b-4e4f-8475-fded6d983294 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 247684:0:(mdd_device.c:990:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 247684:0:(mdd_device.c:990:mdd_trash_setup()) LBUG
CPU: 0 PID: 247684 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
? osd_xattr_get+0x439/0x560 [osd_ldiskfs]
mdd_prepare+0x4c1/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xac6/0xc00 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_niduuid_destroy+0x7c/0x100 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0x7fe/0x17c0 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fe45450f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem fef56f09-897a-4e93-864e-34e98df25f48.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
LustreError: lustre-MDT0000: not available for connect from 10.240.39.250@tcp (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: Skipped 173 previous similar messages
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem fef56f09-897a-4e93-864e-34e98df25f48 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem fef56f09-897a-4e93-864e-34e98df25f48.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 2c324328-92c9-4d18-a78e-a4e2e99d75c6 r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 2c324328-92c9-4d18-a78e-a4e2e99d75c6.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 2c324328-92c9-4d18-a78e-a4e2e99d75c6 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 2c324328-92c9-4d18-a78e-a4e2e99d75c6.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 2c324328-92c9-4d18-a78e-a4e2e99d75c6 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 204614:0:(mdd_device.c:990:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 204614:0:(mdd_device.c:990:mdd_trash_setup()) LBUG
CPU: 0 PID: 204614 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.ddn1.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
? osd_xattr_get+0x439/0x560 [osd_ldiskfs]
mdd_prepare+0x4c1/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xac6/0xc00 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_niduuid_destroy+0x7c/0x100 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0x7fe/0x17c0 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? __mod_memcg_state+0x55/0xa0
? srso_alias_return_thunk+0x5/0xfbef5
? rcu_nocb_try_bypass+0x5e/0x460
? srso_alias_return_thunk+0x5/0xfbef5
? obj_cgroup_uncharge_pages+0x4d/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
? __dentry_kill+0x12b/0x170
? srso_alias_return_thunk+0x5/0xfbef5
? rcu_nocb_try_bypass+0x5e/0x460
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_file_free_rcu+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0x117/0x2b0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f0dfa10f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 6ad5c3e3-e76e-49a5-bc08-2c6d846868ce.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 462f9b54-d691-4325-9014-49e654b880b4.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 6ad5c3e3-e76e-49a5-bc08-2c6d846868ce r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 6ad5c3e3-e76e-49a5-bc08-2c6d846868ce.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 462f9b54-d691-4325-9014-49e654b880b4 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 462f9b54-d691-4325-9014-49e654b880b4.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,raft -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 6ad5c3e3-e76e-49a5-bc08-2c6d846868ce r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64

LustreError: 219526:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 219526:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 219526 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_set_info_async.constprop.0+0x134/0x290 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fcd6c70f13e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 24c69286-6b28-423a-83f2-72534f470ced.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 24c69286-6b28-423a-83f2-72534f470ced r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 24c69286-6b28-423a-83f2-72534f470ced.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem f5dcd233-1790-4698-9b48-ed0a76c8fffd r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem f5dcd233-1790-4698-9b48-ed0a76c8fffd.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem f5dcd233-1790-4698-9b48-ed0a76c8fffd r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem f5dcd233-1790-4698-9b48-ed0a76c8fffd.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem f5dcd233-1790-4698-9b48-ed0a76c8fffd r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 239308:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 239308:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 239308 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_reconnect_import+0x7c/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? mgc_set_info_async+0x5fd/0x670 [mgc]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? folio_add_new_anon_rmap+0x41/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_anonymous_page+0x1bb/0x3e0
? srso_alias_return_thunk+0x5/0xfbef5
? __handle_mm_fault+0x2fe/0x650
? srso_alias_return_thunk+0x5/0xfbef5
? __count_memcg_events+0x4f/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? mm_account_fault+0x6c/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? handle_mm_fault+0x123/0x250
? srso_alias_return_thunk+0x5/0xfbef5
? do_user_addr_fault+0x362/0x620
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f97dbb0f46e

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 7e879e9a-14d6-4abe-9ac6-77321b3ca0a3.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 7e879e9a-14d6-4abe-9ac6-77321b3ca0a3 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ .
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 7e879e9a-14d6-4abe-9ac6-77321b3ca0a3.
Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey
LDISKFS-fs (dm-3): mounted filesystem 8cdbd0b3-c622-4ea3-a287-df3ed9820bba r/w with ordered data mode. Quota mode: journalled.
LDISKFS-fs (dm-3): unmounting filesystem 8cdbd0b3-c622-4ea3-a287-df3ed9820bba.
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 8cdbd0b3-c622-4ea3-a287-df3ed9820bba r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 8cdbd0b3-c622-4ea3-a287-df3ed9820bba.
Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 8cdbd0b3-c622-4ea3-a287-df3ed9820bba r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: reset Object Index mappings

LustreError: 171853:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 171853:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 171853 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? kmem_cache_alloc+0x181/0x340
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? mod_delayed_work_on+0x57/0x90
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? client_connect_import+0x27a/0x5a0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfbef5
? obd_connect.constprop.0+0x8d/0x340 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xf0
? __pfx_file_free_rcu+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0x117/0x2b0
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xf0
? srso_alias_return_thunk+0x5/0xfbef5
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f9fc410f13e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LDISKFS-fs (dm-3): unmounting filesystem 1186f04e-7e36-4001-aa3f-13de8b652696.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem e0c05d2c-c9bc-416b-92d2-331a7cc73588.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 1186f04e-7e36-4001-aa3f-13de8b652696 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 1186f04e-7e36-4001-aa3f-13de8b652696.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem e0c05d2c-c9bc-416b-92d2-331a7cc73588 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem e0c05d2c-c9bc-416b-92d2-331a7cc73588.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 1186f04e-7e36-4001-aa3f-13de8b652696 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages

LustreError: 184000:0:(mdd_device.c:996:mdd_trash_setup()) ASSERTION( ((&mdo->mo_lu)->lo_header->loh_attr & LOHA_EXISTS) ) failed:
LustreError: 184000:0:(mdd_device.c:996:mdd_trash_setup()) LBUG
CPU: 1 PID: 184000 Comm: mount.lustre Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2_lustre.el9.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
mdd_trash_setup+0x21d/0x230 [mdd]
mdd_dot_lustre_setup+0x2b4/0x580 [mdd]
mdd_prepare+0x4b6/0xeb0 [mdd]
? srso_alias_return_thunk+0x5/0xfbef5
mdt_prepare+0x4d/0x3c0 [mdt]
server_start_targets+0xa8c/0xc60 [ptlrpc]
? __pfx_class_config_llog_handler+0x10/0x10 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kick_pool+0x65/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? __queue_work+0x111/0x390
? srso_alias_return_thunk+0x5/0xfbef5
? __slab_free+0xcb/0x310
? srso_alias_return_thunk+0x5/0xfbef5
? ptlrpc_pinger_add_import+0x183/0x240 [ptlrpc]
? lustre_start_mgc+0xa4d/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? kfree+0x2a6/0x330
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_mgc+0xb1b/0x1430 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
? lustre_start_simple+0x78/0x1e0 [obdclass]
server_fill_super+0x64a/0x790 [ptlrpc]
lustre_fill_super+0x38e/0x480 [lustre]
? __pfx_lustre_fill_super+0x10/0x10 [lustre]
mount_nodev+0x41/0x90
legacy_get_tree+0x24/0x50
vfs_get_tree+0x22/0xd0
? srso_alias_return_thunk+0x5/0xfbef5
do_new_mount+0x17a/0x310
__x64_sys_mount+0x107/0x140
do_syscall_64+0x5c/0xe0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? xas_load+0x3d/0x50
? srso_alias_return_thunk+0x5/0xfbef5
? xa_load+0x70/0xb0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? fsnotify_grab_connector+0x49/0x80
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? rcutree_enqueue+0x23/0x140
? srso_alias_return_thunk+0x5/0xfbef5
? __call_rcu_common.constprop.0+0xa7/0x2e0
? srso_alias_return_thunk+0x5/0xfbef5
? wait_task_zombie+0x14a/0x580
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? remove_wait_queue+0x20/0x60
? srso_alias_return_thunk+0x5/0xfbef5
? srso_alias_return_thunk+0x5/0xfbef5
? do_wait+0x9f/0x100
? srso_alias_return_thunk+0x5/0xfbef5
? kernel_wait4+0xc7/0x140
? __pfx_child_wait_callback+0x10/0x10
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_work+0x103/0x130
? srso_alias_return_thunk+0x5/0xfbef5
? syscall_exit_to_user_mode+0x19/0x40
? srso_alias_return_thunk+0x5/0xfbef5
? do_syscall_64+0x6b/0xe0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f137d70f46e

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n osd*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 182131:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup
LustreError: 182131:0:(obd_class.h:478:obd_check_dev()) Skipped 117 previous similar messages
LDISKFS-fs (dm-3): unmounting filesystem 2b82d2ce-78bf-412f-b6f4-3a9f4f088f48.
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds3
LustreError: MGC10.240.45.200@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 3 previous similar messages
Lustre: Failing over lustre-MDT0002
LDISKFS-fs (dm-4): unmounting filesystem 0013af96-05b1-423c-8c6c-c3611459db7b.
Lustre: server umount lustre-MDT0002 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-3): mounted filesystem 2b82d2ce-78bf-412f-b6f4-3a9f4f088f48 r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-3): unmounting filesystem 2b82d2ce-78bf-412f-b6f4-3a9f4f088f48.
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt
Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds3_flakey /mnt/lustre-brpt
LDISKFS-fs (dm-4): mounted filesystem 0013af96-05b1-423c-8c6c-c3611459db7b r/w with ordered data mode. Quota mode: journalled.
Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/oi.16.0
Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt
LDISKFS-fs (dm-4): unmounting filesystem 0013af96-05b1-423c-8c6c-c3611459db7b.
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub,notcu /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem 2b82d2ce-78bf-412f-b6f4-3a9f4f088f48 r/w with ordered data mode. Quota mode: journalled.
Lustre: lustre-MDT0000: invalid oi count 63, remove them, then set it to 64
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 6 previous similar messages