Editing crashreport #74576

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( cp != ((void *)0) ) failedmgs_revoke_lockmgs_revoke_lock
mgs_notify
mgc_process_log
mgc_requeue_thread
kthread
ret_from_fork
11

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
Module load
LustreError: 15605:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15605:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 0 PID: 15605 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60e/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm25.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm24.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm25.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm24.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.42.66@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_24_g905f992
LNet: Added LNI 10.240.42.65@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm24.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm24.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm25.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm25.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm24.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm24.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm25.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm25.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: cli-lustre-OST0006-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:6:ost]
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm23.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm22.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm22.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Autotest: Test running for 5 minutes (lustre-reviews_review-dne-selinux-ssk-part-1_122938.61)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15607:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15607:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 0 PID: 15607 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60e/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm23.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm22.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm23.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm22.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.26@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_24_g905f992
LNet: Added LNI 10.240.31.25@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm22.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm22.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm23.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm23.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm22.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm22.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm23.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm23.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm21.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm20.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15587:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15587:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 1 PID: 15587 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60e/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm129.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm128.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm129.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.174@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_24_g60f2c91
LNet: Added LNI 10.240.47.173@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm128.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm128.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm129.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm129.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm128.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm128.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm129.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm129.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0006-super: Allocated super-sequence [0x0000000480000400-0x00000004c0000400]:6:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm127.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm126.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm126.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15611:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15611:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 0 PID: 15611 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60e/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm167.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm168.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm167.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-155vm168.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.31.171@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_24_g60f2c91
LNet: Added LNI 10.240.31.170@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm167.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm167.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm168.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm168.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm167.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm167.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm168.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm168.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0005: Not available for connect from 10.240.31.172@tcp (not set up)
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm166.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm165.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-155vm165.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15612:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15612:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 1 PID: 15612 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x23a/0x3e0 [mgs]
mgc_process_log+0x7e9/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm174.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm173.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm174.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm173.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_25_g97c88d5
LNet: Added LNI 10.240.24.20@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: cli-lustre-OST0006-super: Allocated super-sequence [0x0000000480000400-0x00000004c0000400]:6:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: hostname -I
Lustre: DEBUG MARKER: hostname -I
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm171.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm171.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15622:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15622:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 0 PID: 15622 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x23a/0x3e0 [mgs]
mgc_process_log+0x7e9/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm173.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm174.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm173.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: onyx-156vm174.onyx.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.24.21@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_25_g97c88d5
LNet: Added LNI 10.240.24.20@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm173.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm174.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm172.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: hostname -I
Lustre: DEBUG MARKER: hostname -I
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm171.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: onyx-156vm171.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15582:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15582:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 1 PID: 15582 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x23a/0x3e0 [mgs]
mgc_process_log+0x7e9/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm109.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm108.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm109.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm108.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.102@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_25_g97c88d5
LNet: Added LNI 10.240.44.101@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm108.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm108.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm109.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm109.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm108.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm108.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm109.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm109.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: cli-lustre-OST0007-super: Allocated super-sequence [0x00000004c0000400-0x0000000500000400]:7:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm107.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: hostname -I
Lustre: DEBUG MARKER: hostname -I
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm106.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm106.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15582:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15582:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 1 PID: 15582 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60e/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm86.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm85.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm85.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-156vm86.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.46.187@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_24_g5919d3b
LNet: Added LNI 10.240.46.186@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0006-super: Allocated super-sequence [0x0000000480000400-0x00000004c0000400]:6:ost]
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-156vm83.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-156vm83.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Link to test
Module load
LustreError: 15595:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15595:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 0 PID: 15595 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60e/0xea0 [mgc]
mgc_requeue_thread+0x343/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xea0/0xea0 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm86.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm85.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm86.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-152vm85.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.47.131@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.51_24_g5919d3b
LNet: Added LNI 10.240.47.130@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm85.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm86.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: cli-lustre-OST0006-super: Allocated super-sequence [0x0000000480000400-0x00000004c0000400]:6:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm84.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-152vm83.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-152vm83.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Link to test
Module load
LustreError: 15510:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15510:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 1 PID: 15510 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60f/0xe60 [mgc]
mgc_requeue_thread+0x33d/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xe60/0xe60 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm82.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm81.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-154vm82.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-154vm81.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.40.49@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.50_144_gc56a941
LNet: Added LNI 10.240.40.48@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm81.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm81.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm82.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm82.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm81.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm81.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm82.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm82.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x00000003c0000400-0x0000000400000400]:3:ost]
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm80.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: cli-lustre-OST0007-super: Allocated super-sequence [0x00000004c0000400-0x0000000500000400]:7:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-154vm79.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-154vm79.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
LustreError: 33708:0:(obd_config.c:1644:class_process_config()) no device for:
Lustre: 33708:0:(obd_config.c:1733:class_process_config()) Ignoring error -22 on optional command 0xce016
Link to test
Module load
LustreError: 15521:0:(mgs_handler.c:234:mgs_revoke_lock()) ASSERTION( cp != ((void *)0) ) failed:
LustreError: 15521:0:(mgs_handler.c:234:mgs_revoke_lock()) LBUG
CPU: 0 PID: 15521 Comm: ll_cfg_requeue Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.89.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
mgs_revoke_lock+0x158/0x330 [mgs]
mgs_notify+0x1c8/0x3c0 [mgs]
mgc_process_log+0x60f/0xe60 [mgc]
mgc_requeue_thread+0x33d/0xae0 [mgc]
? srso_alias_return_thunk+0x5/0xfcdfd
? finish_wait+0x80/0x80
? mgc_process_log+0xe60/0xe60 [mgc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
libcfs: loading out-of-tree module taints kernel.
libcfs: module verification failed: signature and/or required key missing - tainting kernel
Key type ._llcrypt registered
Key type .llcrypt registered
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm112.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm113.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm112.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: trevis-153vm113.trevis.whamcloud.com: executing set_hostid
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost1 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost1' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=0 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost2 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost2' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=1 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost3 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost3' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=2 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost4 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost4' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=3 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost4
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost5 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost5' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=4 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost6 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost6' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=5 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost7 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost7' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=6 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: [ -e /dev/vg_Role_OSS/ost8 ]
Lustre: DEBUG MARKER: grep -c /mnt/lustre-ost8' ' /proc/mounts || true
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: mkfs.lustre --mgsnode=10.240.44.106@tcp --fsname=lustre --ost --index=7 --param=sys.timeout=20 --backfstype=ldiskfs --device-size=8388608 --mkfsoptions="-b 4096" --reformat /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost1' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost2' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost3' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost4' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost5' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost6' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost7' ' /proc/mounts);
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-ost8' ' /proc/mounts);
Lustre: DEBUG MARKER: /usr/sbin/lsvcgssd -vvv -s -m -o -z
Lustre: DEBUG MARKER: num=0;
|| ((num++))
Lustre: DEBUG MARKER: /usr/sbin/l_getauth -d
Lustre: DEBUG MARKER: echo /etc/request-key.d/lgssc.conf: ; cat /etc/request-key.d/lgssc.conf
Lustre: DEBUG MARKER: cp /tmp/test-framework-keys/lustre.key /tmp/test-framework-keys/lustre_cli.key && /usr/sbin/lgss_sk -t client -m /tmp/test-framework-keys/lustre_cli.key >/dev/null 2>&1
Lustre: DEBUG MARKER: modprobe ptlrpc_gss && /usr/sbin/lctl set_param -n sptlrpc.gss.lgss_keyring.debug_level=4
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1
alg: No test for adler32 (adler32-zlib)
Lustre: Lustre: Build Version: 2.17.50_144_gc56a941
LNet: Added LNI 10.240.44.105@tcp [8/256/0/180]
Key type lgssc registered
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gss.rsi_upcall=/usr/sbin/l_getauth
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm112.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm112.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm113.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm113.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm112.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm112.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm113.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm113.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost1
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost1 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost1_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost1 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost1_flakey /mnt/lustre-ost1
Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt'
LDISKFS-fs (dm-9): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0000: new disk, initializing
Lustre: srv-lustre-OST0000: No data found on store. Initialize space.
Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0000-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:0:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost1_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost2_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost2
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost2 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost2_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost2 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost2; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost2_flakey /mnt/lustre-ost2
LDISKFS-fs (dm-10): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0001: new disk, initializing
Lustre: srv-lustre-OST0001: No data found on store. Initialize space.
Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0001-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:1:ost]
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost2_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost2
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost3
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost3 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost3_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost3 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost3; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost3_flakey /mnt/lustre-ost3
LDISKFS-fs (dm-11): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0002: new disk, initializing
Lustre: srv-lustre-OST0002: No data found on store. Initialize space.
Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0002-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost3_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost3
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost4_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost4
Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:2:ost]
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost4 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost4_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost4 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost4; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost4_flakey /mnt/lustre-ost4
LDISKFS-fs (dm-12): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0003: new disk, initializing
Lustre: srv-lustre-OST0003: No data found on store. Initialize space.
Lustre: lustre-OST0003: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0003-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost4_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost4
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost5_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost5
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost5 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost5_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost5 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost5; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost5_flakey /mnt/lustre-ost5
LDISKFS-fs (dm-13): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0004: new disk, initializing
Lustre: srv-lustre-OST0004: No data found on store. Initialize space.
Lustre: lustre-OST0004: Imperative Recovery not enabled, recovery window 60-180
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0004-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: cli-lustre-OST0004-super: Allocated super-sequence [0x0000000400000400-0x0000000440000400]:4:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost5_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost5
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost6_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost6
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost6 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost6_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost6 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost6; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost6_flakey /mnt/lustre-ost6
LDISKFS-fs (dm-14): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0005-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0005-super: Allocated super-sequence [0x0000000440000400-0x0000000480000400]:5:ost]
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost6_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost6
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost7_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost7
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost7 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost7_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost7 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost7; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost7_flakey /mnt/lustre-ost7
LDISKFS-fs (dm-15): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: lustre-OST0006: new disk, initializing
Lustre: Skipped 1 previous similar message
Lustre: srv-lustre-OST0006: No data found on store. Initialize space.
Lustre: Skipped 1 previous similar message
Lustre: lustre-OST0006: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0006-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost7_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost7
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/ost8_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/vg_Role_OSS/ost8
Lustre: DEBUG MARKER: blockdev --getsz /dev/vg_Role_OSS/ost8 2>/dev/null
Lustre: DEBUG MARKER: dmsetup create ost8_flakey --table "0 16777216 linear /dev/vg_Role_OSS/ost8 0"
Lustre: DEBUG MARKER: dmsetup mknodes >/dev/null 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-ost8; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/ost8_flakey /mnt/lustre-ost8
LDISKFS-fs (dm-16): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param seq.cli-lustre-OST0007-super.width=16384
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/utils/gss:/usr/li
Lustre: cli-lustre-OST0007-super: Allocated super-sequence [0x00000004c0000400-0x0000000500000400]:7:ost]
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm111.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: sync; sleep 1; sync
Lustre: DEBUG MARKER: e2label /dev/mapper/ost8_flakey 2>/dev/null
Lustre: DEBUG MARKER: fstrim -v /mnt/lustre-ost8
Lustre: DEBUG MARKER: keyctl show | grep lustre | cut -c1-11 |
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm105.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: trevis-153vm105.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20
Lustre: DEBUG MARKER: Using TIMEOUT=20
Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0
Lustre: DEBUG MARKER: sysctl --values kernel/kptr_restrict
Lustre: DEBUG MARKER: sysctl -wq kernel/kptr_restrict=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=ski
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=ski
LustreError: 33737:0:(obd_config.c:1644:class_process_config()) no device for:
Lustre: 33737:0:(obd_config.c:1733:class_process_config()) Ignoring error -22 on optional command 0xce016
Link to test
Return to new crashes list