Editing crashreport #69264

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( lqe->lqe_is_global ) failedqmt_id_lock_notifyqmt_id_lock_notify
qmt_site_recalc_cb
cfs_hash_for_each_tight
qmt_pool_recalc
kthread
ret_from_fork
6

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanity-quota test 3c: Quota pools: check block soft limit on different pools
LustreError: 102665:0:(qmt_lock.c:1009:qmt_id_lock_notify()) ASSERTION( lqe->lqe_is_global ) failed:
LustreError: 102665:0:(qmt_lock.c:1009:qmt_id_lock_notify()) LBUG
CPU: 0 PID: 102665 Comm: qsd_reint_qpool Kdump: loaded Tainted: G OE --------- - - 4.18.0-477.27.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
qmt_id_lock_notify+0x1ee/0x330 [lquota]
qmt_site_recalc_cb+0x378/0x5b0 [lquota]
? qmt_pool_lqes_lookup_spec+0x430/0x430 [lquota]
cfs_hash_for_each_tight+0x122/0x310 [libcfs]
qmt_pool_recalc+0x33d/0xa30 [lquota]
? __schedule+0x2d9/0x870
? qmt_lgd_extend_cb+0xd0/0xd0 [lquota]
? qmt_lgd_extend_cb+0xd0/0xd0 [lquota]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ugp
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0000-0001/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool2
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 2>/dev/null || echo foo
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool2 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool2 lustre-OST[0000-0001/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 |
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool2 |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after timer goes off
Lustre: DEBUG MARKER: Write after timer goes off
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after cancel lru locks
Lustre: DEBUG MARKER: Write after cancel lru locks
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID
Link to test
sanity-quota test 3c: Quota pools: check block soft limit on different pools
LustreError: 83872:0:(qmt_lock.c:1014:qmt_id_lock_notify()) ASSERTION( lqe->lqe_is_global ) failed:
LustreError: 83872:0:(qmt_lock.c:1014:qmt_id_lock_notify()) LBUG
Pid: 83872, comm: qsd_reint_qpool 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Wed Nov 8 23:01:53 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x63/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] qmt_id_lock_notify+0x1ee/0x330 [lquota]
[<0>] qmt_site_recalc_cb+0x34b/0x550 [lquota]
[<0>] cfs_hash_for_each_tight+0x122/0x310 [libcfs]
[<0>] qmt_pool_recalc+0x33d/0xa20 [lquota]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0-1/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool2
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool2 lustre-OST[0-1/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after timer goes off
Lustre: DEBUG MARKER: Write after timer goes off
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after cancel lru locks
Lustre: DEBUG MARKER: Write after cancel lru locks
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param catastrophe 2>&1
Lustre: DEBUG MARKER: dmesg
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --help
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID
Link to test
sanity-quota test 3c: Quota pools: check block soft limit on different pools
LustreError: 117362:0:(qmt_lock.c:974:qmt_id_lock_notify()) ASSERTION( lqe->lqe_is_global ) failed:
LustreError: 117362:0:(qmt_lock.c:974:qmt_id_lock_notify()) LBUG
Pid: 117362, comm: qsd_reint_qpool 4.18.0-477.27.1.el8_lustre.ddn17.x86_64 #1 SMP Sat Oct 21 17:57:56 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] qmt_id_lock_notify+0x1ee/0x330 [lquota]
[<0>] qmt_site_recalc_cb+0x344/0x540 [lquota]
[<0>] cfs_hash_for_each_tight+0x122/0x310 [libcfs]
[<0>] qmt_pool_recalc+0x372/0x9f0 [lquota]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ugp
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0000-0001/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool2
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool2 lustre-OST[0000-0001/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after timer goes off
Lustre: DEBUG MARKER: Write after timer goes off
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after cancel lru locks
Lustre: DEBUG MARKER: Write after cancel lru locks
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null
Lustre: DEBUG MARKER: lfs --list-commands
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID
Link to test
sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota)
LustreError: 2418364:0:(qmt_lock.c:968:qmt_id_lock_notify()) ASSERTION( lqe->lqe_is_global ) failed:
LustreError: 2418364:0:(qmt_lock.c:968:qmt_id_lock_notify()) LBUG
Pid: 2418364, comm: qsd_reint_qpool 4.18.0-477.21.1.el8_lustre.x86_64 #1 SMP Sat Sep 23 17:41:02 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] qmt_id_lock_notify+0x1ee/0x330 [lquota]
[<0>] qmt_site_recalc_cb+0x34b/0x550 [lquota]
[<0>] cfs_hash_for_each_tight+0x122/0x310 [libcfs]
[<0>] qmt_pool_recalc+0x375/0xa80 [lquota]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ugp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(block hardlimit:20 MB\)
Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0000-0007/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0002-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: dmesg
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write...
Lustre: DEBUG MARKER: Write...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ...
Lustre: DEBUG MARKER: Write out of block quota ...
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark --------------------------------------
Lustre: DEBUG MARKER: --------------------------------------
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(block hardlimit:20 MB\)
Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB)
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write...
Lustre: DEBUG MARKER: Write...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ...
Lustre: DEBUG MARKER: Write out of block quota ...
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark --------------------------------------
Lustre: DEBUG MARKER: --------------------------------------
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Project quota \(block hardlimit:20 mb\)
Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write...
Lustre: DEBUG MARKER: Write...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ...
Lustre: DEBUG MARKER: Write out of block quota ...
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0002_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0003_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0004_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0005_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0006_UUID
Link to test
sanity-quota test 1b: Quota pools: Block hard limit (normal use and out of quota)
LustreError: 2235487:0:(qmt_lock.c:968:qmt_id_lock_notify()) ASSERTION( lqe->lqe_is_global ) failed:
LustreError: 2235487:0:(qmt_lock.c:968:qmt_id_lock_notify()) LBUG
Pid: 2235487, comm: qsd_reint_qpool 4.18.0-477.15.1.el8_lustre.x86_64 #1 SMP Fri Sep 1 20:56:46 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] qmt_id_lock_notify+0x1ee/0x330 [lquota]
[<0>] qmt_site_recalc_cb+0x34b/0x550 [lquota]
[<0>] cfs_hash_for_each_tight+0x122/0x310 [libcfs]
[<0>] qmt_pool_recalc+0x375/0xa80 [lquota]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ugp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark User quota \(block hardlimit:20 MB\)
Lustre: DEBUG MARKER: User quota (block hardlimit:20 MB)
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0000-0006/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: dmesg
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write...
Lustre: DEBUG MARKER: Write...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ...
Lustre: DEBUG MARKER: Write out of block quota ...
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark --------------------------------------
Lustre: DEBUG MARKER: --------------------------------------
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Group quota \(block hardlimit:20 MB\)
Lustre: DEBUG MARKER: Group quota (block hardlimit:20 MB)
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write...
Lustre: DEBUG MARKER: Write...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ...
Lustre: DEBUG MARKER: Write out of block quota ...
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl mark --------------------------------------
Lustre: DEBUG MARKER: --------------------------------------
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Project quota \(block hardlimit:20 mb\)
Lustre: DEBUG MARKER: Project quota (block hardlimit:20 mb)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write...
Lustre: DEBUG MARKER: Write...
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write out of block quota ...
Lustre: DEBUG MARKER: Write out of block quota ...
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0002_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0003_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0004_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0005_UUID
Link to test
sanity-quota test 3c: Quota pools: check block soft limit on different pools
LustreError: 87286:0:(qmt_lock.c:968:qmt_id_lock_notify()) ASSERTION( lqe->lqe_is_global ) failed:
LustreError: 87286:0:(qmt_lock.c:968:qmt_id_lock_notify()) LBUG
Pid: 87286, comm: qsd_reint_qpool 4.18.0-477.15.1.el8_lustre.x86_64 #1 SMP Fri Sep 1 20:56:46 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] qmt_id_lock_notify+0x1ee/0x330 [lquota]
[<0>] qmt_site_recalc_cb+0x34b/0x550 [lquota]
[<0>] cfs_hash_for_each_tight+0x122/0x310 [libcfs]
[<0>] qmt_pool_recalc+0x375/0xa80 [lquota]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ugp
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool1 lustre-OST[0000-0001/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 |
Lustre: DEBUG MARKER: lctl pool_new lustre.qpool2
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_add lustre.qpool2 lustre-OST[0000-0001/1]
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool2 |
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n qmt.lustre-QMT0000.dt-0x0.soft_least_qunit
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param fail_val=0 fail_loc=0
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after timer goes off
Lustre: DEBUG MARKER: Write after timer goes off
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Write after cancel lru locks
Lustre: DEBUG MARKER: Write after cancel lru locks
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n os[cd]*.*MD*.force_sync 1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl set_param -n os[cd]*.*MDT*.force_sync=1
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool1
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0000_UUID
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool1 lustre-OST0001_UUID
Lustre: DEBUG MARKER: lctl pool_list lustre.qpool1 | wc -l
Lustre: DEBUG MARKER: lctl pool_destroy lustre.qpool1
Lustre: DEBUG MARKER: lctl get_param -n lod.lustre-MDT0000-mdtlov.pools.qpool1 2>/dev/null || echo foo
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre
Lustre: DEBUG MARKER: /usr/sbin/lctl pool_list lustre.qpool2
Lustre: DEBUG MARKER: lctl pool_remove lustre.qpool2 lustre-OST0000_UUID
Link to test
Return to new crashes list