Editing crashreport #73144

ReasonCrashing FunctionWhere to cut BacktraceReports Count
BUG: unable to handle kernel paging requestllog_osd_next_blockllog_origin_handle_next_block
tgt_llog_next_block
tgt_request_handle
ptlrpc_server_handle_request
ptlrpc_main
kthread
ret_from_fork
4

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):
Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):
Limit to a test:
(Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Failures list (last 100):

Failing TestFull CrashMessages before crashComment
sanity test 253: Check object allocation limit
BUG: unable to handle kernel paging request at ffff8d16f19bd220
PGD 61c01067 P4D 61c01067 PUD 0
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 379836 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE --------- - - 4.18.0-372.9.1.el8_lustre.ddn17.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:llog_osd_next_block+0x7cc/0xff0 [obdclass]
Code: f8 48 89 4c 24 30 e8 03 c6 fe ff 48 8b 4c 24 30 48 8b 54 24 38 85 c0 0f 85 c9 05 00 00 8b 01 48 29 c2 48 8b 04 24 48 8d 34 10 <8b> 46 08 25 ff f0 00 00 3d 10 60 00 00 0f 84 1d 02 00 00 8b 51 04
RSP: 0018:ffffb9984185fc38 EFLAGS: 00010287
RAX: ffff8d1701fdc218 RBX: 0000000000001000 RCX: ffff8d1701fdd210
RDX: ffffffffef9e1000 RSI: ffff8d16f19bd218 RDI: ffff8d1708c6e000
RBP: ffff8d1701fdc210 R08: 000000000000fd00 R09: ffff8d1701fdd218
R10: 0000000000000001 R11: 0000000000000246 R12: 00000000000001b4
R13: ffff8d1701fdc208 R14: ffff8d17683c7a00 R15: ffff8d1708c6e000
FS: 0000000000000000(0000) GS:ffff8d17bfd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff8d16f19bd220 CR3: 0000000060210001 CR4: 00000000000606e0
Call Trace:
llog_origin_handle_next_block+0x5dc/0x7b0 [ptlrpc]
tgt_llog_next_block+0x2b/0xb0 [ptlrpc]
tgt_request_handle+0xc90/0x1970 [ptlrpc]
ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc]
ptlrpc_main+0xba6/0x14a0 [ptlrpc]
? __schedule+0x2d9/0x830
? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc]
kthread+0x10a/0x120
? set_kthread_struct+0x40/0x40
ret_from_fork+0x35/0x40
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev i2c_piix4 pcspkr virtio_balloon sunrpc dm_mod ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net serio_raw net_failover failover virtio_blk [last unloaded: obdecho]
CR2: ffff8d16f19bd220
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*OST*-osc-MDT0000.reserved_mb_high
Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl pool_new lustre.test_253
LustreError: 524135:0:(osd_io.c:2107:osd_ldiskfs_write_record()) lustre-MDT0000: error reading offset 430080 (block 105, size 224, offs 430080), credits 33/33: rc = -28
LustreError: 524135:0:(osd_io.c:2107:osd_ldiskfs_write_record()) Skipped 27 previous similar messages
LustreError: 524135:0:(mgs_handler.c:971:mgs_iocontrol_pool()) OBD_IOC_POOL err -28, cmd CE020 for pool lustre.test_253
Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_253: @@@@@@ FAIL: No pool created, result code 28
Lustre: DEBUG MARKER: sanity test_253: @@@@@@ FAIL: No pool created, result code 28
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-1/2025-05-03/lustre-b_es6_0_rolling-upgrade-downgrade_792_216_49e867da-15f1-43dc-9381-223144f1bdd7/lustre-b_es6_0_rolling-downgrade-mds_792_216_8//sanity.test_253.debug_log.$(hostname -s).1746425972.log;
LustreError: 379836:0:(llog_osd.c:1043:llog_osd_next_block()) lustre-MDT0000-osd: invalid llog tail at log id [0x3:0xa:0x0]:0 offset 430080 last_rec idx 512 tail idx 2 lrt len 1721 read_size 4096
Lustre: DEBUG MARKER: lctl set_param -n fail_loc=0 fail_val=0 2>/dev/null
Link to test
sanity test 253: Check object allocation limit
BUG: unable to handle kernel paging request at ffff95ccf5521220
PGD 97a01067 P4D 97a01067 PUD 0
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 373589 Comm: ll_mgs_0000 Kdump: loaded Tainted: G OE --------- - - 4.18.0-372.9.1.el8_lustre.ddn17.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:llog_osd_next_block+0x7cc/0xff0 [obdclass]
Code: f8 48 89 4c 24 30 e8 03 c6 fe ff 48 8b 4c 24 30 48 8b 54 24 38 85 c0 0f 85 c9 05 00 00 8b 01 48 29 c2 48 8b 04 24 48 8d 34 10 <8b> 46 08 25 ff f0 00 00 3d 10 60 00 00 0f 84 1d 02 00 00 8b 51 04
RSP: 0018:ffffa695c16b3c38 EFLAGS: 00010287
RAX: ffff95cd05b40218 RBX: 0000000000001000 RCX: ffff95cd05b41210
RDX: ffffffffef9e1000 RSI: ffff95ccf5521218 RDI: ffff95cd0af80400
RBP: ffff95cd05b40210 R08: 000000000000fd00 R09: ffff95cd05b41218
R10: 0000000000000001 R11: 0000000000000000 R12: 00000000000001b4
R13: ffff95cd05b40208 R14: ffff95cd89473c00 R15: ffff95cd0af80400
FS: 0000000000000000(0000) GS:ffff95cdbfd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff95ccf5521220 CR3: 0000000096010004 CR4: 00000000000606e0
Call Trace:
llog_origin_handle_next_block+0x5dc/0x7b0 [ptlrpc]
tgt_llog_next_block+0x2b/0xb0 [ptlrpc]
tgt_request_handle+0xc90/0x1970 [ptlrpc]
ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc]
ptlrpc_main+0xba6/0x14a0 [ptlrpc]
? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc]
kthread+0x10a/0x120
? set_kthread_struct+0x40/0x40
ret_from_fork+0x35/0x40
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon joydev i2c_piix4 sunrpc pcspkr dm_mod ext4 mbcache jbd2 ata_generic ata_piix libata virtio_net crc32c_intel net_failover serio_raw virtio_blk failover [last unloaded: obdecho]
CR2: ffff95ccf5521220
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*OST*-osc-MDT0000.reserved_mb_high
Lustre: DEBUG MARKER: /usr/sbin/lctl list_param osp.*osc*.old_sync_processed 2> /dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*osc*.old_sync_processed
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osc.*MDT*.sync_*
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osp.*.destroys_in_flight
Lustre: DEBUG MARKER: lctl pool_new lustre.test_253
LustreError: 517809:0:(mgs_handler.c:971:mgs_iocontrol_pool()) OBD_IOC_POOL err -28, cmd CE020 for pool lustre.test_253
Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_253: @@@@@@ FAIL: No pool created, result code 28
Lustre: DEBUG MARKER: sanity test_253: @@@@@@ FAIL: No pool created, result code 28
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-1/2025-05-03/lustre-b_es6_0_rolling-upgrade-downgrade_792_214_ce2903b4-7a6d-4b5f-a21d-829895049b68/lustre-b_es6_0_rolling-downgrade-mds_792_214_8//sanity.test_253.debug_log.$(hostname -s).1746420279.log;
LustreError: 373590:0:(llog_osd.c:1043:llog_osd_next_block()) lustre-MDT0000-osd: invalid llog tail at log id [0x3:0xa:0x0]:0 offset 430080 last_rec idx 7761772 tail idx 2 lrt len 1666 read_size 4096
LustreError: 373593:0:(mgc_request.c:613:do_requeue()) failed processing log: -22
LustreError: 518012:0:(qmt_pool.c:986:qmt_pool_new()) lustre-QMT0000: can't prepare pool for test_253: rc = -28
Link to test
sanity test 224c: Don't hang if one of md lost during large bulk RPC
BUG: unable to handle kernel paging request at ffff9ca9faa59220
PGD 5c001067 P4D 5c001067 PUD 0
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 381117 Comm: ll_mgs_0000 Kdump: loaded Tainted: G OE --------- - - 4.18.0-372.9.1.el8_lustre.ddn17.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:llog_osd_next_block+0x7cc/0xff0 [obdclass]
Code: f8 48 89 4c 24 30 e8 03 c6 fe ff 48 8b 4c 24 30 48 8b 54 24 38 85 c0 0f 85 c9 05 00 00 8b 01 48 29 c2 48 8b 04 24 48 8d 34 10 <8b> 46 08 25 ff f0 00 00 3d 10 60 00 00 0f 84 1d 02 00 00 8b 51 04
RSP: 0018:ffffacb5c17dbc38 EFLAGS: 00010287
RAX: ffff9caa0b078218 RBX: 0000000000001000 RCX: ffff9caa0b079210
RDX: ffffffffef9e1000 RSI: ffff9ca9faa59218 RDI: ffff9caa8186f800
RBP: ffff9caa0b078210 R08: 000000000000fd00 R09: ffff9caa0b079218
R10: 0000000000000001 R11: 0000000000000100 R12: 00000000000001b4
R13: ffff9caa0b078208 R14: ffff9caa2ce6fc00 R15: ffff9caa8186f800
FS: 0000000000000000(0000) GS:ffff9cab3bd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff9ca9faa59220 CR3: 000000005a610002 CR4: 00000000003706e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
llog_origin_handle_next_block+0x5dc/0x7b0 [ptlrpc]
tgt_llog_next_block+0x2b/0xb0 [ptlrpc]
tgt_request_handle+0xc90/0x1970 [ptlrpc]
ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc]
ptlrpc_main+0xba6/0x14a0 [ptlrpc]
? __schedule+0x2d9/0x830
? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc]
kthread+0x10a/0x120
? set_kthread_struct+0x40/0x40
ret_from_fork+0x35/0x40
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr i2c_piix4 virtio_balloon sunrpc dm_mod ext4 mbcache jbd2 ata_generic crc32c_intel ata_piix serio_raw virtio_net libata virtio_blk net_failover failover [last unloaded: dm_flakey]
CR2: ffff9ca9faa59220
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.at_max='0'
LustreError: 502150:0:(llog_osd.c:1043:llog_osd_next_block()) lustre-MDT0000-osd: invalid llog tail at log id [0x3:0xa:0x0]:0 offset 430080 last_rec idx 0 tail idx 2 lrt len 1655 read_size 4096
LustreError: 502150:0:(llog_osd.c:1043:llog_osd_next_block()) Skipped 9 previous similar messages
LustreError: 502149:0:(mgs_llog.c:967:mgs_modify()) MGS: modify lustre/sys.at_max failed: rc = -22
LustreError: 502149:0:(mgs_llog.c:2027:mgs_write_log_direct_all()) MGS: Can't modify llog lustre-client: rc = -22
LustreError: 502149:0:(mgs_llog.c:2035:mgs_write_log_direct_all()) MGS: writing log lustre-client: rc = -28
LustreError: 502149:0:(mgs_llog.c:2035:mgs_write_log_direct_all()) Skipped 1 previous similar message
LustreError: 502157:0:(llog_osd.c:1055:llog_osd_next_block()) lustre-MDT0000-osd: invalid llog tail at log id [0xa:0xa:0x0]:0 offset 86016 bytes 4096
Link to test
sanity test 224c: Don't hang if one of md lost during large bulk RPC
BUG: unable to handle kernel paging request at ffff8d7d72811220
PGD ae801067 P4D ae801067 PUD 0
Oops: 0000 [#1] SMP PTI
CPU: 1 PID: 381213 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE --------- - - 4.18.0-372.9.1.el8_lustre.ddn17.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
RIP: 0010:llog_osd_next_block+0x7cc/0xff0 [obdclass]
Code: f8 48 89 4c 24 30 e8 03 c6 fe ff 48 8b 4c 24 30 48 8b 54 24 38 85 c0 0f 85 c9 05 00 00 8b 01 48 29 c2 48 8b 04 24 48 8d 34 10 <8b> 46 08 25 ff f0 00 00 3d 10 60 00 00 0f 84 1d 02 00 00 8b 51 04
RSP: 0018:ffffa4da817b3c38 EFLAGS: 00010287
RAX: ffff8d7d82e30218 RBX: 0000000000001000 RCX: ffff8d7d82e31210
RDX: ffffffffef9e1000 RSI: ffff8d7d72811218 RDI: ffff8d7dccb5ec00
RBP: ffff8d7d82e30210 R08: 000000000000fd00 R09: ffff8d7d82e31218
R10: 0000000000000001 R11: ffff8d7dea8530cc R12: 00000000000001b4
R13: ffff8d7d82e30208 R14: ffff8d7df8cb8200 R15: ffff8d7dccb5ec00
FS: 0000000000000000(0000) GS:ffff8d7e3cd00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: ffff8d7d72811220 CR3: 00000000ace10002 CR4: 00000000001706e0
Call Trace:
llog_origin_handle_next_block+0x5dc/0x7b0 [ptlrpc]
tgt_llog_next_block+0x2b/0xb0 [ptlrpc]
tgt_request_handle+0xc90/0x1970 [ptlrpc]
ptlrpc_server_handle_request+0x323/0xbc0 [ptlrpc]
ptlrpc_main+0xba6/0x14a0 [ptlrpc]
? ptlrpc_register_service+0xfb0/0xfb0 [ptlrpc]
kthread+0x10a/0x120
? set_kthread_struct+0x40/0x40
ret_from_fork+0x35/0x40
Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev virtio_balloon pcspkr i2c_piix4 sunrpc dm_mod ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net net_failover virtio_blk failover serio_raw [last unloaded: dm_flakey]
CR2: ffff8d7d72811220
Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.sys.at_max='0'
LustreError: 502008:0:(llog_osd.c:1043:llog_osd_next_block()) lustre-MDT0000-osd: invalid llog tail at log id [0x3:0xa:0x0]:0 offset 430080 last_rec idx 1937075200 tail idx 2 lrt len 1677 read_size 4096
LustreError: 502008:0:(llog_osd.c:1043:llog_osd_next_block()) Skipped 9 previous similar messages
LustreError: 502005:0:(mgs_llog.c:967:mgs_modify()) MGS: modify lustre/sys.at_max failed: rc = -22
LustreError: 502005:0:(mgs_llog.c:2027:mgs_write_log_direct_all()) MGS: Can't modify llog lustre-client: rc = -22
LustreError: 502005:0:(mgs_llog.c:2035:mgs_write_log_direct_all()) MGS: writing log lustre-client: rc = -28
LustreError: 502005:0:(mgs_llog.c:2035:mgs_write_log_direct_all()) Skipped 1 previous similar message
LustreError: 502014:0:(llog_osd.c:1055:llog_osd_next_block()) lustre-MDT0000-osd: invalid llog tail at log id [0xa:0xa:0x0]:0 offset 86016 bytes 4096
LustreError: 502005:0:(mgs_llog.c:4325:mgs_write_log_param()) err -28 on param 'sys.at_max=0'
LustreError: 502005:0:(mgs_handler.c:1021:mgs_iocontrol()) MGS: setparam err: rc = -28
Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity test_224c: @@@@@@ FAIL: \/usr\/sbin\/lctl conf_param lustre.sys.at_max failed
Lustre: DEBUG MARKER: sanity test_224c: @@@@@@ FAIL: /usr/sbin/lctl conf_param lustre.sys.at_max failed
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-1/2025-04-25/lustre-b_es6_0_rolling-upgrade-downgrade_791_214_8e408d17-1d3d-43dc-8ab5-ddcd367c2765/lustre-b_es6_0_rolling-downgrade-mds_791_214_8//sanity.test_224c.debug_log.$(hostname -s).1745701851.log;
Link to test
Return to new crashes list