| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| conf-sanity testing /usr/lib64/lustre/tests/disk2_10-ldiskfs.tar.bz2 upgrade with writeconf | LustreError: 479734:0:(mdt_handler.c:4221:mdt_lock_handle_fini()) ASSERTION( !lustre_handle_is_used(&lh->mlh_reg_lh) ) failed: LustreError: 479734:0:(mdt_handler.c:4221:mdt_lock_handle_fini()) LBUG Pid: 479734, comm: mdt00_002 4.18.0-553.53.1.el8_lustre.x86_64 #1 SMP Sun Jul 13 21:07:27 UTC 2025 Call Trace TBD: [<0>] libcfs_call_trace+0x63/0x90 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] mdt_lock_handle_fini+0x48/0x80 [mdt] [<0>] mdt_thread_info_fini+0xbb/0x170 [mdt] [<0>] mdt_reint+0x68/0x110 [mdt] [<0>] tgt_request_handle+0xccd/0x1a20 [ptlrpc] [<0>] ptlrpc_server_handle_request+0x23f/0xc60 [ptlrpc] [<0>] ptlrpc_main+0xbfa/0x1540 [ptlrpc] [<0>] kthread+0x134/0x150 [<0>] ret_from_fork+0x1f/0x40 | Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: mkdir -p /tmp/t32/mnt/mdt /tmp/t32/mnt/mdt1 /tmp/t32/mnt/ost /tmp/t32/mnt/ost1 Lustre: DEBUG MARKER: tar xjvf /usr/lib64/lustre/tests/disk2_10-ldiskfs.tar.bz2 -S -C /tmp/t32 Lustre: DEBUG MARKER: cat /tmp/t32/commit Lustre: DEBUG MARKER: cat /tmp/t32/kernel Lustre: DEBUG MARKER: cat /tmp/t32/arch Lustre: DEBUG MARKER: cat /tmp/t32/bspace Lustre: DEBUG MARKER: cat /tmp/t32/ispace Lustre: DEBUG MARKER: test -f /tmp/t32/blimit Lustre: DEBUG MARKER: cat /tmp/t32/blimit Lustre: DEBUG MARKER: test -f /tmp/t32/ilimit Lustre: DEBUG MARKER: cat /tmp/t32/ilimit Lustre: DEBUG MARKER: test -f /tmp/t32/mdt2 Lustre: DEBUG MARKER: test -f /tmp/t32/ost2 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/mdt2 Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt loop0: detected capacity change from 0 to 204800000 Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/mdt2 loop0: detected capacity change from 0 to 204800000 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm190.onyx.whamcloud.com: executing \/usr\/sbin\/lctl device_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm190.onyx.whamcloud.com: executing \/usr\/sbin\/lctl device_list Lustre: DEBUG MARKER: onyx-155vm190.onyx.whamcloud.com: executing /usr/sbin/lctl device_list Lustre: DEBUG MARKER: onyx-155vm190.onyx.whamcloud.com: executing /usr/sbin/lctl device_list Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm190.onyx.whamcloud.com: executing losetup -a Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm190.onyx.whamcloud.com: executing losetup -a Lustre: DEBUG MARKER: onyx-155vm190.onyx.whamcloud.com: executing losetup -a Lustre: DEBUG MARKER: onyx-155vm190.onyx.whamcloud.com: executing losetup -a Lustre: DEBUG MARKER: mount -t lustre -o loop,writeconf /tmp/t32/mdt /tmp/t32/mnt/mdt loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: t32fs-MDT0000-osd: enabled 'large_dir' feature on device /dev/loop0 Lustre: Skipped 1 previous similar message Lustre: 479708:0:(scrub.c:191:scrub_file_load()) t32fs-MDT0000: reset scrub OI count for format change (LU-16655) Lustre: 479708:0:(scrub.c:191:scrub_file_load()) Skipped 1 previous similar message Lustre: MGS: Logs for fs t32fs were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 Lustre: Setting parameter t32fs-MDT0000-mdtlov.lov.stripesize in log t32fs-MDT0000 Lustre: t32fs-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: mount -t lustre -o mgsnode=10.240.31.194@tcp,loop,writeconf /tmp/t32/mdt2 /tmp/t32/mnt/mdt1 loop1: detected capacity change from 0 to 204800000 LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: MGS: Regenerating t32fs-MDT0001 log by user request: rc = 0 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n mdt.t32fs*.enable_remote_dir=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.t32fs-MDT0000.uuid Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost Lustre: DEBUG MARKER: tunefs.lustre --dryrun /tmp/t32/ost2 Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost loop2: detected capacity change from 0 to 204800000 Lustre: DEBUG MARKER: tunefs.lustre --quota /tmp/t32/ost2 loop2: detected capacity change from 0 to 204800000 Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.240.31.194@tcp,writeconf /tmp/t32/ost /tmp/t32/mnt/ost loop2: detected capacity change from 0 to 204800000 LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: mount -t lustre -onomgs -oloop,mgsnode=10.240.31.194@tcp,writeconf /tmp/t32/ost2 /tmp/t32/mnt/ost1 loop3: detected capacity change from 0 to 204800000 LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc Lustre: MGS: Regenerating t32fs-OST0001 log by user request: rc = 0 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0000.uuid Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n obdfilter.t32fs-OST0001.uuid Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.failover.node=10.240.31.194@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.failover.node=10.240.31.194@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-OST0000.osc.max_dirty_mb=15 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdc.max_rpcs_in_flight=9 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.lov.stripesize=4M Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0000.mdd.atime_diff=70 Lustre: DEBUG MARKER: /usr/sbin/lctl pool_new t32fs.interop Lustre: t32fs-OST0000: deleting orphan objects from 0x280000400:68 to 0x280000400:129 Lustre: t32fs-OST0001: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:129 Lustre: t32fs-OST0000: deleting orphan objects from 0x0:11 to 0x0:97 Lustre: t32fs-OST0001: deleting orphan objects from 0x0:10 to 0x0:97 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0001.failover.node=10.240.31.194@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0001.mdc.max_rpcs_in_flight=9 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs-MDT0001.lov.stripesize=4M Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.mdt='ug' Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param t32fs.quota.ost='ug' Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled LustreError: 483909:0:(qsd_reint.c:635:qqi_reint_delayed()) t32fs-MDT0000: Delaying reintegration for qtype:0 until pending updates are flushed. LustreError: 483909:0:(qsd_reint.c:635:qqi_reint_delayed()) Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.t32fs-OST0000.quota_slave.enabled Lustre: DEBUG MARKER: test -f /tmp/t32/list Lustre: DEBUG MARKER: test -f /tmp/t32/list2 Lustre: DEBUG MARKER: cat /tmp/t32/list2 LustreError: 479738:0:(ldlm_lockd.c:1257:ldlm_handle_enqueue0()) @@@ invalid lock request flags 1000000 req@000000008772606b x1845883918415168/t0(0) o101->t32fs-MDT0000-mdtlov_UUID@0@lo:727/0 lens 328/0 e 0 to 0 dl 1760372317 ref 1 fl Interpret:/0/ffffffff rc 0/-1 job:'' LustreError: 11-0: t32fs-MDT0001-osp-MDT0000: operation ldlm_enqueue to node 0@lo failed: rc = -71 | Link to test |