Editing crashreport #74176

ReasonCrashing FunctionWhere to cut BacktraceReports Count
ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed__osc_lru_del__osc_lru_del
osc_lru_use
osc_page_cache_add
osc_io_commit_async
cl_io_commit_async
lov_io_commit_async
cl_io_commit_async
vvp_io_fault_start
cl_io_start
cl_io_loop
ll_page_mkwrite0
ll_page_mkwrite
do_page_mkwrite
do_pte_missing
__handle_mm_fault
handle_mm_fault
do_user_addr_fault
exc_page_fault
asm_exc_page_fault
10

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanity-flr test 70a: flr mode fsx test
LustreError: 670272:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 670272:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 0 PID: 670272 Comm: fsx Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.13.1.el9_7.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x50 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d1/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xab4/0xcd0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x16e/0x4c0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
do_page_mkwrite+0x4e/0xa0
do_pte_missing+0x8e/0x200
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x620
exc_page_fault+0x61/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7fe974f91fef
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 641840:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 641840:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 0 PID: 641840 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d4/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xac3/0xce0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x169/0x4b0 [lustre]
do_page_mkwrite+0x4c/0xe0
do_pte_missing+0x6f/0x190
__handle_mm_fault+0x2fb/0x690
? finish_task_switch.isra.0+0x8c/0x2a0
handle_mm_fault+0xdc/0x270
do_user_addr_fault+0x1b4/0x6a0
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7f6046f9062f
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 635352:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 635352:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 1 PID: 635352 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d4/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xac3/0xce0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x169/0x4b0 [lustre]
do_page_mkwrite+0x4c/0xe0
do_pte_missing+0x6f/0x190
__handle_mm_fault+0x2fb/0x690
handle_mm_fault+0xdc/0x270
do_user_addr_fault+0x1b4/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7f17dc790628
Lustre: 310585:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff252b2e973689c0 x1853792488825728/t0(0) o10->lustre-OST0000-osc-ff252b2e890d5000@10.240.47.199@tcp:6/4 lens 440/432 e 0 to 0 dl 1767924455 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 310585:0:(client.c:1636:after_reply()) Skipped 3 previous similar messages
Lustre: 310586:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff252b2ece8e6a40 x1853792488849408/t0(0) o10->lustre-OST0000-osc-ff252b2e890d5000@10.240.47.199@tcp:6/4 lens 440/432 e 0 to 0 dl 1767924466 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 310586:0:(client.c:1636:after_reply()) Skipped 9 previous similar messages
Lustre: 310585:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff252b2e97b7e080 x1853792488973440/t0(0) o10->lustre-OST0000-osc-ff252b2e890d5000@10.240.47.199@tcp:6/4 lens 440/432 e 0 to 0 dl 1767924475 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 310585:0:(client.c:1636:after_reply()) Skipped 2 previous similar messages
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 650459:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 650459:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 0 PID: 650459 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d1/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xab4/0xcd0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x16f/0x4c0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
do_page_mkwrite+0x4e/0xa0
do_pte_missing+0x8e/0x200
__handle_mm_fault+0x2fe/0x650
? srso_alias_return_thunk+0x5/0xfbef5
handle_mm_fault+0xe9/0x250
do_user_addr_fault+0x218/0x620
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7f71be190ae2
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 634712:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 634712:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 1 PID: 634712 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d4/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xac3/0xce0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x169/0x4b0 [lustre]
do_page_mkwrite+0x4c/0xe0
do_pte_missing+0x6f/0x190
__handle_mm_fault+0x2fb/0x690
? rseq_ip_fixup+0x6e/0x1a0
handle_mm_fault+0xdc/0x270
do_user_addr_fault+0x1b4/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7faa1179062f
Lustre: 310671:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff3fba65176849c0 x1851112244975232/t0(0) o10->lustre-OST0003-osc-ff3fba6503dbe000@10.240.23.119@tcp:6/4 lens 440/432 e 0 to 0 dl 1765369121 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 665692:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 665692:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 1 PID: 665692 Comm: fsx Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.9.1.el9_7.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x50 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d1/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xab4/0xcd0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x16e/0x4c0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
do_page_mkwrite+0x4e/0xa0
do_pte_missing+0x8e/0x200
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xfb/0x270
do_user_addr_fault+0x215/0x620
exc_page_fault+0x61/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7f0f3fd91fe8
Lustre: 327277:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff32f892d96cea40 x1851111732004864/t0(0) o10->lustre-OST0002-osc-ff32f89322744000@10.240.46.174@tcp:6/4 lens 440/432 e 0 to 0 dl 1765368193 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 327277:0:(client.c:1636:after_reply()) Skipped 1 previous similar message
Lustre: 327277:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff32f892e0fda700 x1851111732436224/t0(0) o10->lustre-OST0000-osc-ff32f89322744000@10.240.46.174@tcp:6/4 lens 440/432 e 0 to 0 dl 1765368201 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 327277:0:(client.c:1636:after_reply()) Skipped 6 previous similar messages
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 649767:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 649767:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 1 PID: 649767 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d1/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xab4/0xcd0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x16f/0x4c0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
do_page_mkwrite+0x4e/0xa0
do_pte_missing+0x8e/0x200
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xe9/0x250
do_user_addr_fault+0x218/0x620
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7fb6fed90aef
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 648380:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 648380:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 0 PID: 648380 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.62.1.el9_6.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d1/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xab4/0xcd0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x16f/0x4c0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
do_page_mkwrite+0x4e/0xa0
do_pte_missing+0x8e/0x200
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xe9/0x250
do_user_addr_fault+0x218/0x620
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7fd001d90ae8
Lustre: 317776:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff2e777942013400 x1850395470849024/t0(0) o10->lustre-OST0004-osc-ff2e7779640ed000@10.240.46.175@tcp:6/4 lens 440/432 e 0 to 0 dl 1764684702 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 637421:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 637421:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 0 PID: 637421 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-2.el9_5.1 04/01/2014
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d4/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x7e/0x160 [obdclass]
vvp_io_fault_start+0xac3/0xce0 [lustre]
? srso_alias_return_thunk+0x5/0xfbef5
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x5e/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? srso_alias_return_thunk+0x5/0xfbef5
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x169/0x4b0 [lustre]
do_page_mkwrite+0x4c/0xe0
do_pte_missing+0x6f/0x190
__handle_mm_fault+0x2fb/0x690
? srso_alias_return_thunk+0x5/0xfbef5
handle_mm_fault+0xdc/0x270
do_user_addr_fault+0x1b4/0x6a0
? srso_alias_return_thunk+0x5/0xfbef5
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7fdba6f90636
Lustre: 312557:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff4c96781aedc680 x1850395382951680/t0(0) o10->lustre-OST0005-osc-ff4c967832bff800@10.240.39.240@tcp:6/4 lens 440/432 e 0 to 0 dl 1764684643 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 312557:0:(client.c:1636:after_reply()) Skipped 2 previous similar messages
Lustre: 312557:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff4c9677c375ad80 x1850395382971392/t0(0) o10->lustre-OST0005-osc-ff4c967832bff800@10.240.39.240@tcp:6/4 lens 440/432 e 0 to 0 dl 1764684649 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 312557:0:(client.c:1636:after_reply()) Skipped 5 previous similar messages
Lustre: 312556:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ff4c9677c3758000 x1850395382989824/t0(0) o10->lustre-OST0006-osc-ff4c967832bff800@10.240.39.240@tcp:6/4 lens 440/432 e 0 to 0 dl 1764684659 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 312556:0:(client.c:1636:after_reply()) Skipped 10 previous similar messages
Autotest: Test running for 295 minutes (lustre-master_full-part-1_4677.166)
Link to test
sanity-flr test 70a: flr mode fsx test
LustreError: 1070182:0:(osc_page.c:470:__osc_lru_del()) ASSERTION( atomic_long_read(&cli->cl_lru_in_list) >= 0 ) failed:
LustreError: 1070182:0:(osc_page.c:470:__osc_lru_del()) LBUG
CPU: 1 PID: 1070182 Comm: fsx Kdump: loaded Tainted: G OE ------- --- 5.14.0-570.42.2.el9_6.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
__osc_lru_del+0xaa/0xb0 [osc]
osc_lru_use.part.0+0x41/0x70 [osc]
osc_page_cache_add+0x88/0x180 [osc]
osc_io_commit_async+0x2d1/0x520 [osc]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x81/0x160 [obdclass]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
lov_io_commit_async+0x149/0x5b0 [lov]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
? __pfx_mkwrite_commit_callback+0x10/0x10 [lustre]
cl_io_commit_async+0x81/0x160 [obdclass]
vvp_io_fault_start+0xab4/0xcd0 [lustre]
? cl_lock_request+0x66/0x1f0 [obdclass]
cl_io_start+0x61/0x130 [obdclass]
cl_io_loop+0x99/0x220 [obdclass]
? _raw_spin_unlock_irq+0xa/0x30
ll_page_mkwrite0.constprop.0+0x101/0x670 [lustre]
ll_page_mkwrite+0x16f/0x4c0 [lustre]
do_page_mkwrite+0x51/0xa0
do_pte_missing+0x8e/0x200
__handle_mm_fault+0x2fe/0x650
handle_mm_fault+0xe9/0x250
do_user_addr_fault+0x218/0x620
exc_page_fault+0x62/0x150
asm_exc_page_fault+0x22/0x30
RIP: 0033:0x7fa532aa412e
Lustre: 687886:0:(client.c:1636:after_reply()) @@@ resending request on EINPROGRESS req@ffff9dc5847a3740 x1849327562808576/t0(0) o10->lustre-OST0000-osc-ffff9dc496579800@10.240.41.94@tcp:6/4 lens 440/432 e 0 to 0 dl 1763656794 ref 1 fl Rpc:RQU/602/0 rc 0/-115 job:'fsx.0' uid:0 gid:0 projid:0
Lustre: 687886:0:(client.c:1636:after_reply()) Skipped 1 previous similar message
Link to test
Return to new crashes list