Edit new crash report

Reason	Crashing Function	Where to cut Backtrace	Reports Count
LBUG	llog_osd_write_rec	llog_osd_write_rec llog_write_rec llog_cat_new_log llog_cat_add_rec llog_add osp_sync_add_rec osp_sync_add osp_destroy lod_sub_destroy lod_obj_for_each_stripe lod_destroy mdd_finish_unlink mdd_unlink mdt_reint_unlink mdt_reint_rec mdt_reint_internal mdt_reint tgt_request_handle ptlrpc_server_handle_request ptlrpc_main kthread ret_from_fork	4

Failing Test	Full Crash	Messages before crash	Comment
sanity test 136: Race catalog processing 2	LustreError: 933586:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG Pid: 933586, comm: mdt00_001 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Sat Dec 23 04:44:29 UTC 2023 Call Trace TBD: [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] llog_osd_write_rec+0x1b50/0x1b80 [obdclass] [<0>] llog_write_rec+0x3fb/0x530 [obdclass] [<0>] llog_cat_new_log+0x4d5/0xe90 [obdclass] [<0>] llog_cat_add_rec+0x10b/0x930 [obdclass] [<0>] llog_add+0x19d/0x1f0 [obdclass] [<0>] osp_sync_add_rec+0x37e/0x6f0 [osp] [<0>] osp_sync_add+0x69/0x70 [osp] [<0>] osp_destroy+0x171/0x2c0 [osp] [<0>] lod_sub_destroy+0x222/0x4b0 [lod] [<0>] lod_obj_for_each_stripe+0x118/0x2e0 [lod] [<0>] lod_destroy+0x6b8/0x790 [lod] [<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd] [<0>] mdd_unlink+0x706/0xd60 [mdd] [<0>] mdt_reint_unlink+0xbac/0x1320 [mdt] [<0>] mdt_reint_rec+0x11f/0x270 [mdt] [<0>] mdt_reint_internal+0x4d0/0x7e0 [mdt] [<0>] mdt_reint+0x5d/0x110 [mdt] [<0>] tgt_request_handle+0xccd/0x1b10 [ptlrpc] [<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc] [<0>] ptlrpc_main+0xc0f/0x1560 [ptlrpc] [<0>] kthread+0x134/0x150 [<0>] ret_from_fork+0x35/0x40	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 933608:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 933608:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 1 previous similar message Autotest: Test running for 135 minutes (lustre-b2_15_rolling-upgrade-client1-zfs_82.123) Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 933608:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0 Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0 Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 276 previous similar messages Lustre: 933587:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0 Lustre: 933587:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 1116 previous similar messages Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0 Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 2237 previous similar messages Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0 Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 1004 previous similar messages Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0 Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 7456 previous similar messages LustreError: 933586:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 40579 already set in llog bitmap [0x1:0x2:0x0]	Link to test
sanity test 136: Race catalog processing 2	LustreError: 343644:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG Pid: 343644, comm: mdt00_000 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Sat Dec 23 04:44:29 UTC 2023 Call Trace TBD: [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] llog_osd_write_rec+0x1b50/0x1b80 [obdclass] [<0>] llog_write_rec+0x3fb/0x530 [obdclass] [<0>] llog_cat_new_log+0x4d5/0xe90 [obdclass] [<0>] llog_cat_add_rec+0x10b/0x930 [obdclass] [<0>] llog_add+0x19d/0x1f0 [obdclass] [<0>] osp_sync_add_rec+0x37e/0x6f0 [osp] [<0>] osp_sync_add+0x69/0x70 [osp] [<0>] osp_destroy+0x171/0x2c0 [osp] [<0>] lod_sub_destroy+0x222/0x4b0 [lod] [<0>] lod_obj_for_each_stripe+0x118/0x2e0 [lod] [<0>] lod_destroy+0x6b8/0x790 [lod] [<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd] [<0>] mdd_unlink+0x706/0xd60 [mdd] [<0>] mdt_reint_unlink+0xbac/0x1320 [mdt] [<0>] mdt_reint_rec+0x11f/0x270 [mdt] [<0>] mdt_reint_internal+0x4d0/0x7e0 [mdt] [<0>] mdt_reint+0x5d/0x110 [mdt] [<0>] tgt_request_handle+0xccd/0x1b10 [ptlrpc] [<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc] [<0>] ptlrpc_main+0xc0f/0x1560 [ptlrpc] [<0>] kthread+0x134/0x150 [<0>] ret_from_fork+0x35/0x40	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 2 previous similar messages Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 1 previous similar message Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 6 previous similar messages Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 13 previous similar messages Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 28 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 24858923 previous similar messages LustreError: 343644:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 60964 already set in llog bitmap [0x1:0x2:0x0]	Link to test
sanity test 136: Race catalog processing 2	LustreError: 332752:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG Pid: 332752, comm: mdt00_000 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Sat Dec 23 04:44:29 UTC 2023 Call Trace TBD: [<0>] libcfs_call_trace+0x6f/0xa0 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] llog_osd_write_rec+0x1b50/0x1b80 [obdclass] [<0>] llog_write_rec+0x3fb/0x530 [obdclass] [<0>] llog_cat_new_log+0x4d5/0xe90 [obdclass] [<0>] llog_cat_add_rec+0x10b/0x930 [obdclass] [<0>] llog_add+0x19d/0x1f0 [obdclass] [<0>] osp_sync_add_rec+0x37e/0x6f0 [osp] [<0>] osp_sync_add+0x69/0x70 [osp] [<0>] osp_destroy+0x171/0x2c0 [osp] [<0>] lod_sub_destroy+0x222/0x4b0 [lod] [<0>] lod_obj_for_each_stripe+0x118/0x2e0 [lod] [<0>] lod_destroy+0x6b8/0x790 [lod] [<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd] [<0>] mdd_unlink+0x706/0xd60 [mdd] [<0>] mdt_reint_unlink+0xbac/0x1320 [mdt] [<0>] mdt_reint_rec+0x11f/0x270 [mdt] [<0>] mdt_reint_internal+0x4d0/0x7e0 [mdt] [<0>] mdt_reint+0x5d/0x110 [mdt] [<0>] tgt_request_handle+0xccd/0x1b10 [ptlrpc] [<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc] [<0>] ptlrpc_main+0xc0f/0x1560 [ptlrpc] [<0>] kthread+0x134/0x150 [<0>] ret_from_fork+0x35/0x40	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 1 previous similar message Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 30385801 previous similar messages LustreError: 332752:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 55725 already set in llog bitmap [0x1:0x2:0x0]	Link to test
sanity test 136: Race catalog processing 2	LustreError: 379505:0:(llog_osd.c:631:llog_osd_write_rec()) LBUG Pid: 379505, comm: mdt00_000 4.18.0-372.9.1.el8_lustre.ddn17.x86_64 #1 SMP Mon Jul 18 21:29:58 UTC 2022 Call Trace TBD: [<0>] libcfs_call_trace+0x6f/0x90 [libcfs] [<0>] lbug_with_loc+0x3f/0x70 [libcfs] [<0>] llog_osd_write_rec+0x1a18/0x1a20 [obdclass] [<0>] llog_write_rec+0x456/0x500 [obdclass] [<0>] llog_cat_new_log+0x4e4/0xdf0 [obdclass] [<0>] llog_cat_add_rec+0x108/0x880 [obdclass] [<0>] llog_add+0x196/0x1e0 [obdclass] [<0>] osp_sync_add_rec+0x34d/0x6e0 [osp] [<0>] osp_sync_add+0x69/0x70 [osp] [<0>] osp_destroy+0x171/0x290 [osp] [<0>] lod_sub_destroy+0x218/0x480 [lod] [<0>] lod_obj_for_each_stripe+0x11f/0x2b0 [lod] [<0>] lod_destroy+0x641/0x720 [lod] [<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd] [<0>] mdd_unlink+0x6c2/0xbe0 [mdd] [<0>] mdt_reint_unlink+0xd43/0x1570 [mdt] [<0>] mdt_reint_rec+0x11f/0x250 [mdt] [<0>] mdt_reint_internal+0x498/0x780 [mdt] [<0>] mdt_reint+0x5e/0x100 [mdt] [<0>] tgt_request_handle+0xc90/0x1970 [ptlrpc] [<0>] ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc] [<0>] ptlrpc_main+0xba6/0x14a0 [ptlrpc] [<0>] kthread+0x10a/0x120 [<0>] ret_from_fork+0x35/0x40	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 1 previous similar message Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 1 previous similar message Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 3 previous similar messages Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 4 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug="trace super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck" Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0 Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 6 previous similar messages Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 59997648 previous similar messages LustreError: 379505:0:(llog_osd.c:629:llog_osd_write_rec()) lustre-MDT0000-osd: index 63393 already set in llog bitmap [0x1:0x2:0x0]	Link to test

Failing Test

Full Crash

Messages before crash

Comment

sanity test 136: Race catalog processing 2

LustreError: 933586:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG
Pid: 933586, comm: mdt00_001 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Sat Dec 23 04:44:29 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] llog_osd_write_rec+0x1b50/0x1b80 [obdclass]
[<0>] llog_write_rec+0x3fb/0x530 [obdclass]
[<0>] llog_cat_new_log+0x4d5/0xe90 [obdclass]
[<0>] llog_cat_add_rec+0x10b/0x930 [obdclass]
[<0>] llog_add+0x19d/0x1f0 [obdclass]
[<0>] osp_sync_add_rec+0x37e/0x6f0 [osp]
[<0>] osp_sync_add+0x69/0x70 [osp]
[<0>] osp_destroy+0x171/0x2c0 [osp]
[<0>] lod_sub_destroy+0x222/0x4b0 [lod]
[<0>] lod_obj_for_each_stripe+0x118/0x2e0 [lod]
[<0>] lod_destroy+0x6b8/0x790 [lod]
[<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd]
[<0>] mdd_unlink+0x706/0xd60 [mdd]
[<0>] mdt_reint_unlink+0xbac/0x1320 [mdt]
[<0>] mdt_reint_rec+0x11f/0x270 [mdt]
[<0>] mdt_reint_internal+0x4d0/0x7e0 [mdt]
[<0>] mdt_reint+0x5d/0x110 [mdt]
[<0>] tgt_request_handle+0xccd/0x1b10 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]
[<0>] ptlrpc_main+0xc0f/0x1560 [ptlrpc]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 933608:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 933608:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 1 previous similar message
Autotest: Test running for 135 minutes (lustre-b2_15_rolling-upgrade-client1-zfs_82.123)
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck"
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 933608:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0
Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0
Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 276 previous similar messages
Lustre: 933587:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0
Lustre: 933587:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 1116 previous similar messages
Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0
Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 2237 previous similar messages
Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0
Lustre: 933586:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 1004 previous similar messages
Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) lustre-OST0000-osc-MDT0000: there are no more free slots in catalog [0x2:0x1:0x0]:0
Lustre: 1000032:0:(llog_cat.c:101:llog_cat_new_log()) Skipped 7456 previous similar messages
LustreError: 933586:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 40579 already set in llog bitmap [0x1:0x2:0x0]

Link to test

sanity test 136: Race catalog processing 2

LustreError: 343644:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG
Pid: 343644, comm: mdt00_000 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Sat Dec 23 04:44:29 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] llog_osd_write_rec+0x1b50/0x1b80 [obdclass]
[<0>] llog_write_rec+0x3fb/0x530 [obdclass]
[<0>] llog_cat_new_log+0x4d5/0xe90 [obdclass]
[<0>] llog_cat_add_rec+0x10b/0x930 [obdclass]
[<0>] llog_add+0x19d/0x1f0 [obdclass]
[<0>] osp_sync_add_rec+0x37e/0x6f0 [osp]
[<0>] osp_sync_add+0x69/0x70 [osp]
[<0>] osp_destroy+0x171/0x2c0 [osp]
[<0>] lod_sub_destroy+0x222/0x4b0 [lod]
[<0>] lod_obj_for_each_stripe+0x118/0x2e0 [lod]
[<0>] lod_destroy+0x6b8/0x790 [lod]
[<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd]
[<0>] mdd_unlink+0x706/0xd60 [mdd]
[<0>] mdt_reint_unlink+0xbac/0x1320 [mdt]
[<0>] mdt_reint_rec+0x11f/0x270 [mdt]
[<0>] mdt_reint_internal+0x4d0/0x7e0 [mdt]
[<0>] mdt_reint+0x5d/0x110 [mdt]
[<0>] tgt_request_handle+0xccd/0x1b10 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]
[<0>] ptlrpc_main+0xc0f/0x1560 [ptlrpc]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 2 previous similar messages
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 1 previous similar message
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 6 previous similar messages
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 13 previous similar messages
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 28 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck"
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 343667:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 24858923 previous similar messages
LustreError: 343644:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 60964 already set in llog bitmap [0x1:0x2:0x0]

Link to test

sanity test 136: Race catalog processing 2

LustreError: 332752:0:(llog_osd.c:630:llog_osd_write_rec()) LBUG
Pid: 332752, comm: mdt00_000 4.18.0-477.27.1.el8_lustre.x86_64 #1 SMP Sat Dec 23 04:44:29 UTC 2023
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0xa0 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] llog_osd_write_rec+0x1b50/0x1b80 [obdclass]
[<0>] llog_write_rec+0x3fb/0x530 [obdclass]
[<0>] llog_cat_new_log+0x4d5/0xe90 [obdclass]
[<0>] llog_cat_add_rec+0x10b/0x930 [obdclass]
[<0>] llog_add+0x19d/0x1f0 [obdclass]
[<0>] osp_sync_add_rec+0x37e/0x6f0 [osp]
[<0>] osp_sync_add+0x69/0x70 [osp]
[<0>] osp_destroy+0x171/0x2c0 [osp]
[<0>] lod_sub_destroy+0x222/0x4b0 [lod]
[<0>] lod_obj_for_each_stripe+0x118/0x2e0 [lod]
[<0>] lod_destroy+0x6b8/0x790 [lod]
[<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd]
[<0>] mdd_unlink+0x706/0xd60 [mdd]
[<0>] mdt_reint_unlink+0xbac/0x1320 [mdt]
[<0>] mdt_reint_rec+0x11f/0x270 [mdt]
[<0>] mdt_reint_internal+0x4d0/0x7e0 [mdt]
[<0>] mdt_reint+0x5d/0x110 [mdt]
[<0>] tgt_request_handle+0xccd/0x1b10 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x323/0xbe0 [ptlrpc]
[<0>] ptlrpc_main+0xc0f/0x1560 [ptlrpc]
[<0>] kthread+0x134/0x150
[<0>] ret_from_fork+0x35/0x40

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug="super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck"
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 1 previous similar message
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 332782:0:(llog_cat.c:933:llog_cat_process_or_fork()) Skipped 30385801 previous similar messages
LustreError: 332752:0:(llog_osd.c:628:llog_osd_write_rec()) lustre-MDT0000-osd: index 55725 already set in llog bitmap [0x1:0x2:0x0]

Link to test

sanity test 136: Race catalog processing 2

LustreError: 379505:0:(llog_osd.c:631:llog_osd_write_rec()) LBUG
Pid: 379505, comm: mdt00_000 4.18.0-372.9.1.el8_lustre.ddn17.x86_64 #1 SMP Mon Jul 18 21:29:58 UTC 2022
Call Trace TBD:
[<0>] libcfs_call_trace+0x6f/0x90 [libcfs]
[<0>] lbug_with_loc+0x3f/0x70 [libcfs]
[<0>] llog_osd_write_rec+0x1a18/0x1a20 [obdclass]
[<0>] llog_write_rec+0x456/0x500 [obdclass]
[<0>] llog_cat_new_log+0x4e4/0xdf0 [obdclass]
[<0>] llog_cat_add_rec+0x108/0x880 [obdclass]
[<0>] llog_add+0x196/0x1e0 [obdclass]
[<0>] osp_sync_add_rec+0x34d/0x6e0 [osp]
[<0>] osp_sync_add+0x69/0x70 [osp]
[<0>] osp_destroy+0x171/0x290 [osp]
[<0>] lod_sub_destroy+0x218/0x480 [lod]
[<0>] lod_obj_for_each_stripe+0x11f/0x2b0 [lod]
[<0>] lod_destroy+0x641/0x720 [lod]
[<0>] mdd_finish_unlink+0x8b/0x2a0 [mdd]
[<0>] mdd_unlink+0x6c2/0xbe0 [mdd]
[<0>] mdt_reint_unlink+0xd43/0x1570 [mdt]
[<0>] mdt_reint_rec+0x11f/0x250 [mdt]
[<0>] mdt_reint_internal+0x498/0x780 [mdt]
[<0>] mdt_reint+0x5e/0x100 [mdt]
[<0>] tgt_request_handle+0xc90/0x1970 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc]
[<0>] ptlrpc_main+0xba6/0x14a0 [ptlrpc]
[<0>] kthread+0x10a/0x120
[<0>] ret_from_fork+0x35/0x40

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x131a fail_val=1
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 1 previous similar message
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 1 previous similar message
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 3 previous similar messages
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 4 previous similar messages
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug="trace super ioctl neterror warning dlmtrace error emerg ha rpctrace vfstrace config console lfsck"
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=0
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 6 previous similar messages
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x2:0x1:0x0] crosses index zero
Lustre: 379529:0:(llog_cat.c:930:llog_cat_process_or_fork()) Skipped 59997648 previous similar messages
LustreError: 379505:0:(llog_osd.c:629:llog_osd_write_rec()) lustre-MDT0000-osd: index 63393 already set in llog bitmap [0x1:0x2:0x0]

Link to test

Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below):
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below):
Limit to a test: (Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Editing crashreport #70221

Added fields:

Failures list (last 100):