BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31beca063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm ata_piix drm_panel_orientation_quirks floppy serio_raw libata i2c_core virtio_blk [last unloaded: libcfs]
CPU: 1 PID: 3667 Comm: ptlrpcd_00_15 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802ec4024f0 ti: ffff8802e3ce0000 task.ti: ffff8802e3ce0000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff8802e3ce3930 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff88023f66e470 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff8802e3ce3970 R08: 0000000000000000 R09: ffff8802e3ce3b38
R10: 0000000000000000 R11: 0000000000000001 R12: ffff880331a5bf00
R13: ffffffff81dbbf10 R14: ffff880331a5bf44 R15: 0000000000090000
FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000002e6a0e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa068b52b>] _debug_req+0x8b/0x8f0 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff81068309>] ? kvm_sched_clock_read+0x9/0x20
[<ffffffff81035ec9>] ? sched_clock+0x9/0x10
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff817e3277>] ? _raw_spin_unlock_irqrestore+0x17/0x30
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa06773f6>] ? ptlrpc_set_add_new_req+0xe6/0x160 [ptlrpc]
[<ffffffffa06ac2a4>] ? ptlrpcd_add_req+0x164/0x440 [ptlrpc]
[<ffffffffa0990fb6>] brw_interpret+0xdb6/0xe70 [osc]
[<ffffffffa067ab78>] ptlrpc_check_set+0x3d8/0x2220 [ptlrpc]
[<ffffffffa06ad0f4>] ptlrpcd+0x9c4/0xa80 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa06ac730>] ? ptlrpcd_ctl_init+0x1b0/0x1b0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31be3d063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix serio_raw libata virtio_blk floppy i2c_core [last unloaded: libcfs]
CPU: 7 PID: 12976 Comm: ptlrpcd_00_02 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031d929280 ti: ffff8802d8a5c000 task.ti: ffff8802d8a5c000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff8802d8a5f930 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff8800a2d38570 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff8802d8a5f970 R08: 0000000000000000 R09: ffffffff810baff0
R10: 0000000000000000 R11: 000000000000000f R12: ffff880331bdbf00
R13: ffffffff81dbbf10 R14: ffff880331bdbf44 R15: 0000000000390000
FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000000b9026000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa064452b>] _debug_req+0x8b/0x8f0 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810d0100>] ? try_to_wake_up+0x170/0x390
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff810d03f2>] ? default_wake_function+0x12/0x20
[<ffffffff810bb001>] ? woken_wake_function+0x11/0x20
[<ffffffff810c8510>] ? __wake_up_common+0x70/0x100
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa06303f6>] ? ptlrpc_set_add_new_req+0xe6/0x160 [ptlrpc]
[<ffffffffa06652a4>] ? ptlrpcd_add_req+0x164/0x440 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa0989fb6>] brw_interpret+0xdb6/0xe70 [osc]
[<ffffffffa0633b78>] ptlrpc_check_set+0x3d8/0x2220 [ptlrpc]
[<ffffffffa06660f4>] ptlrpcd+0x9c4/0xa80 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa0665730>] ? ptlrpcd_ctl_init+0x1b0/0x1b0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0
LustreError: lustre-OST0002: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: lustre-OST0003-osc-ffff8800a9864138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x4342:0x0] object 0x300000401:24285 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: lustre-OST0002: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0002-osc-ffff8800a9864138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0002: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 19 previous similar messages
LustreError: lustre-OST0002-osc-ffff8800a9864138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
LustreError: 12987:0:(osc_request.c:2439:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802e5aac640 x1831258367072768/t12884927795(12884927795) o4->lustre-OST0002-osc-ffff8800a9864138@0@lo:6/4 lens 488/448 e 0 to 0 dl 1746426151 ref 2 fl Interpret:RMQU/600/0 rc 0/0 job:'ptlrpcd_00_11.0' uid:0 gid:0 projid:0
LustreError: 12987:0:(osc_request.c:2439:osc_brw_redo_request()) Skipped 9 previous similar messages
Lustre: lustre-OST0000-osc-ffff8800a9864138: disconnect after 20s idle
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x4342:0x0] object 0x300000401:24285 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0002-osc-ffff8800a9864138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x4342:0x0] object 0x300000401:24285 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 5 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 19 previous similar messages
LustreError: lustre-OST0002-osc-ffff8800a9864138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 5 previous similar messages
LustreError: 12975:0:(osc_request.c:2439:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a85bb740 x1831258367087872/t12884927800(12884927800) o4->lustre-OST0002-osc-ffff8800a9864138@0@lo:6/4 lens 488/448 e 0 to 0 dl 1746426186 ref 2 fl Interpret:RMQU/600/0 rc 0/0 job:'ptlrpcd_00_01.0' uid:0 gid:0 projid:0
LustreError: 12975:0:(osc_request.c:2439:osc_brw_redo_request()) Skipped 9 previous similar messages
LustreError: 12978:0:(osc_request.c:2594:brw_interpret()) lustre-OST0002-osc-ffff8800a9864138: too many resent retries for object: 11811161089:24189: rc = -11
LustreError: 12978:0:(osc_request.c:2594:brw_interpret()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: set checksum type to adler, rc = 0
LustreError: lustre-OST0002: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: Skipped 15 previous similar messages
LustreError: lustre-OST0002-osc-ffff8800a9864138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x4342:0x0] object 0x2c0000401:24189 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: Skipped 15 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 47 previous similar messages
LustreError: 12984:0:(osc_request.c:2594:brw_interpret()) lustre-OST0002-osc-ffff8800a9864138: too many resent retries for object: 11811161089:24189: rc = -11
Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0
LustreError: 12976:0:(osc_request.c:2439:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8800a2d3ad40 x1831258367126400/t12884927813(12884927813) o4->lustre-OST0002-osc-ffff8800a9864138@0@lo:6/4 lens 488/448 e 0 to 0 dl 1746426254 ref 2 fl Interpret:RMQU/600/0 rc 0/0 job:'directio.0' uid:0 gid:0 projid:0
LustreError: 12976:0:(osc_request.c:2439:osc_brw_redo_request()) Skipped 21 previous similar messages

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31bf22063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy [last unloaded: libcfs]
CPU: 4 PID: 13082 Comm: ptlrpcd_00_00 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b318010 ti: ffff8800a7e58000 task.ti: ffff8800a7e58000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff8800a7e5b940 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff8802e77dda70 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff8800a7e5b980 R08: ffffffff81813740 R09: ffffffff810baff0
R10: 0000000000000000 R11: 0000000000000400 R12: ffff880331b1bf00
R13: ffffffff81dbbf10 R14: ffff880331b1bf44 R15: 0000000000210000
FS: 0000000000000000(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 0000000001c10000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa06a0720>] _debug_req+0x80/0x8b0 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810d0100>] ? try_to_wake_up+0x170/0x390
[<ffffffff810d03f2>] ? default_wake_function+0x12/0x20
[<ffffffff810bb001>] ? woken_wake_function+0x11/0x20
[<ffffffff810c8510>] ? __wake_up_common+0x70/0x100
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa06c0b40>] ? ptlrpcd_add_req+0x160/0x3f0 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa0942546>] brw_interpret+0xdb6/0xe70 [osc]
[<ffffffffa0690bf8>] ptlrpc_check_set+0x3d8/0x2220 [ptlrpc]
[<ffffffffa06c1a24>] ptlrpcd+0xaa4/0xb80 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa06c0f80>] ? ptlrpcd_ctl_init+0x1b0/0x1b0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0
LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x44f6:0x0] object 0x240000bd0:24978 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: lustre-OST0000-osc-ffff8802b74a8958: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x44f6:0x0] object 0x240000bd0:24978 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 12 previous similar messages
LustreError: lustre-OST0001-osc-ffff8802b74a8958: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
LustreError: 13086:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802e1fac640 x1816933878022784/t12884928466(12884928466) o4->lustre-OST0001-osc-ffff8802b74a8958@0@lo:6/4 lens 488/448 e 0 to 0 dl 1732764972 ref 2 fl Interpret:RMQU/200/0 rc 0/0 job:'ptlrpcd_00_04.0' uid:0 gid:0
LustreError: 13086:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 6 previous similar messages
LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0001-osc-ffff8802b74a8958: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
Lustre: lustre-OST0002-osc-ffff8802b74a8958: disconnect after 20s idle
Lustre: Skipped 3 previous similar messages
LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0001-osc-ffff8802b74a8958: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303]: client csum 8b19b060, server csum 8b19b05f
LustreError: Skipped 3 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 23 previous similar messages
LustreError: lustre-OST0000-osc-ffff8802b74a8958: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x44f6:0x0] object 0x240000bd0:24978 extent [0-4194303], original client csum 8b19b060 (type 1), server csum 8b19b05f (type 1), client csum now 8b19b060
LustreError: Skipped 3 previous similar messages
LustreError: 13097:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802e898e940 x1816933878036608/t21474863022(21474863022) o4->lustre-OST0000-osc-ffff8802b74a8958@0@lo:6/4 lens 488/448 e 0 to 0 dl 1732765006 ref 2 fl Interpret:RMQU/200/0 rc 0/0 job:'ptlrpcd_00_13.0' uid:0 gid:0
LustreError: 13097:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 11 previous similar messages
LustreError: 13087:0:(osc_request.c:2605:brw_interpret()) lustre-OST0001-osc-ffff8802b74a8958: too many resent retries for object: 10737419265:24659: rc = -11
Lustre: DEBUG MARKER: set checksum type to adler, rc = 0
LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x44f6:0x0] object 0x240000bd0:24978 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: Skipped 16 previous similar messages
LustreError: lustre-OST0001-osc-ffff8802b74a8958: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200001b71:0x44f6:0x0] object 0x280000401:24659 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: Skipped 15 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 47 previous similar messages
LustreError: 13087:0:(osc_request.c:2450:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802f6d40540 x1816933878068736/t21474863034(21474863034) o4->lustre-OST0000-osc-ffff8802b74a8958@0@lo:6/4 lens 488/448 e 0 to 0 dl 1732765071 ref 2 fl Interpret:RMQU/200/0 rc 0/0 job:'ptlrpcd_00_03.0' uid:0 gid:0
LustreError: 13087:0:(osc_request.c:2450:osc_brw_redo_request()) Skipped 21 previous similar messages
LustreError: 13091:0:(osc_request.c:2605:brw_interpret()) lustre-OST0001-osc-ffff8802b74a8958: too many resent retries for object: 10737419265:24659: rc = -11
LustreError: 13091:0:(osc_request.c:2605:brw_interpret()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31bfbf063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy virtio_blk libata i2c_core [last unloaded: libcfs]
CPU: 9 PID: 16847 Comm: ptlrpcd_04_00 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880324d45c40 ti: ffff880325408000 task.ti: ffff880325408000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff88032540b940 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff88009e241e70 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff88032540b980 R08: ffffffff818148c0 R09: ffffffff821c2c00
R10: 0000000000000000 R11: 0000000000000400 R12: ffff880331c5bf00
R13: ffffffff81dbbf10 R14: ffff880331c5bf44 R15: 0000000000490000
FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000003205b2000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa0623960>] _debug_req+0x80/0x8b0 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff81068309>] ? kvm_sched_clock_read+0x9/0x20
[<ffffffff81035ec9>] ? sched_clock+0x9/0x10
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff817e3277>] ? _raw_spin_unlock_irqrestore+0x17/0x30
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa0643d60>] ? ptlrpcd_add_req+0x160/0x3f0 [ptlrpc]
[<ffffffffa0960546>] brw_interpret+0xdb6/0xe70 [osc]
[<ffffffffa0613d28>] ptlrpc_check_set+0x3d8/0x2220 [ptlrpc]
[<ffffffffa0644c44>] ptlrpcd+0xaa4/0xb80 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa06441a0>] ? ptlrpcd_ctl_init+0x1b0/0x1b0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: DEBUG MARKER: set checksum type to adler, rc = 0
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: lustre-OST0003-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x2c0000bd0:19772 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0000-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x2c0000bd0:19772 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x2c0000bd0:19772 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0000-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x2c0000bd0:19772 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: Skipped 3 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 21 previous similar messages
LustreError: 16844:0:(osc_request.c:2449:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880085a94b40 x1816165379868416/t12884929587(12884929587) o4->lustre-OST0003-osc-ffff8801d41bc138@0@lo:6/4 lens 488/448 e 0 to 0 dl 1732032384 ref 2 fl Interpret:RMQU/200/0 rc 0/0 job:'ptlrpcd_02_01.0' uid:0 gid:0
LustreError: 16844:0:(osc_request.c:2449:osc_brw_redo_request()) Skipped 10 previous similar messages
LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x2c0000bd0:19772 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: Skipped 3 previous similar messages
LustreError: lustre-OST0000-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x2c0000bd0:19772 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: Skipped 3 previous similar messages
Lustre: lustre-OST0001-osc-ffff8801d41bc138: disconnect after 24s idle
Lustre: Skipped 3 previous similar messages
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303]: client csum 7d33b9a0, server csum 7d33b99f
LustreError: Skipped 5 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 17 previous similar messages
LustreError: lustre-OST0003-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303], original client csum 7d33b9a0 (type 2), server csum 7d33b99f (type 2), client csum now 7d33b9a0
LustreError: Skipped 5 previous similar messages
LustreError: 16843:0:(osc_request.c:2449:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88004dcfc640 x1816165379914368/t12884929592(12884929592) o4->lustre-OST0003-osc-ffff8801d41bc138@0@lo:6/4 lens 488/448 e 0 to 0 dl 1732032419 ref 2 fl Interpret:RMQU/200/0 rc 0/0 job:'ptlrpcd_02_00.0' uid:0 gid:0
LustreError: 16843:0:(osc_request.c:2449:osc_brw_redo_request()) Skipped 8 previous similar messages
LustreError: 16843:0:(osc_request.c:2604:brw_interpret()) lustre-OST0003-osc-ffff8801d41bc138: too many resent retries for object: 15032386565:19387: rc = -11
LustreError: 16849:0:(osc_request.c:2604:brw_interpret()) lustre-OST0000-osc-ffff8801d41bc138: too many resent retries for object: 11811163088:19772: rc = -11
Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303]: client csum de2bf73f, server csum de2bf73e
LustreError: Skipped 16 previous similar messages
LustreError: lustre-OST0003-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303], original client csum de2bf73f (type 4), server csum de2bf73e (type 4), client csum now de2bf73f
LustreError: Skipped 16 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 47 previous similar messages
LustreError: 16848:0:(osc_request.c:2604:brw_interpret()) lustre-OST0000-osc-ffff8801d41bc138: too many resent retries for object: 11811163088:19772: rc = -11
Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0
LustreError: 16848:0:(osc_request.c:2449:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88022b6d8040 x1816165380026368/t12884929605(12884929605) o4->lustre-OST0003-osc-ffff8801d41bc138@0@lo:6/4 lens 488/448 e 0 to 0 dl 1732032487 ref 2 fl Interpret:RMQU/200/0 rc 0/0 job:'directio.0' uid:0 gid:0
LustreError: 16848:0:(osc_request.c:2449:osc_brw_redo_request()) Skipped 21 previous similar messages
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303]: client csum 3a674101, server csum 3a674100
LustreError: Skipped 24 previous similar messages
LustreError: lustre-OST0003-osc-ffff8801d41bc138: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x2000032e1:0x3767:0x0] object 0x380000405:19387 extent [0-4194303], original client csum 3a674101 (type 10), server csum 3a674100 (type 10), client csum now 3a674101
LustreError: Skipped 24 previous similar messages
LustreError: 16848:0:(osc_request.c:2604:brw_interpret()) lustre-OST0003-osc-ffff8801d41bc138: too many resent retries for object: 15032386565:19387: rc = -11
LustreError: 16848:0:(osc_request.c:2604:brw_interpret()) Skipped 1 previous similar message
Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31bf7a063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw libata i2c_core floppy virtio_blk [last unloaded: libcfs]
CPU: 9 PID: 21267 Comm: ptlrpcd_04_00 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88029946c9d0 ti: ffff88027c470000 task.ti: ffff88027c470000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff88027c473948 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff88028c986970 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff88027c473988 R08: 0000000000000000 R09: ffffffff810baff0
R10: 0000000000000000 R11: 000000000000000f R12: ffff880331c5bf00
R13: ffffffff81dbbf10 R14: ffff880331c5bf44 R15: 0000000000490000
FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 0000000292398000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa0623760>] _debug_req+0x80/0x8b0 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810d0100>] ? try_to_wake_up+0x170/0x390
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff810d03f2>] ? default_wake_function+0x12/0x20
[<ffffffff810bb001>] ? woken_wake_function+0x11/0x20
[<ffffffff810c8510>] ? __wake_up_common+0x70/0x100
[<ffffffff814119f9>] ? do_raw_spin_unlock+0x49/0x90
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa0643a80>] ? ptlrpcd_add_req+0x160/0x3f0 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa09684f6>] brw_interpret+0xd66/0xe20 [osc]
[<ffffffffa0613d28>] ptlrpc_check_set+0x3d8/0x2220 [ptlrpc]
[<ffffffffa0644964>] ptlrpcd+0xaa4/0xb80 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa0643ec0>] ? ptlrpcd_ctl_init+0x1b0/0x1b0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: *** cfs_fail_loc=409, val=0***
LustreError: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000013a1:0x5190:0x0] object 0x300000bd0:27632 extent [0-1048575]: client csum 896c0727, server csum 896c0726
LustreError: lustre-OST0003-osc-ffff8803240a2e98: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 0@lo inode [0x2000013a1:0x5190:0x0] object 0x300000bd0:27632 extent [0-1048575], original client csum 896c0727 (type 20), server csum 896c0726 (type 20), client csum now 896c0726
LustreError: 21268:0:(osc_request.c:2449:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802c9b5ee40 x1814444697697280/t17179900914(17179900914) o4->lustre-OST0003-osc-ffff8803240a2e98@0@lo:6/4 lens 488/448 e 0 to 0 dl 1730390611 ref 3 fl Interpret:RQU/204/0 rc 0/0 job:'dd.0' uid:0 gid:0
Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0
Lustre: *** cfs_fail_loc=408, val=0***
LustreError: lustre-OST0003-osc-ffff8803240a2e98: BAD READ CHECKSUM: from 0@lo inode [0x2000013a1:0x5190:0x0] object 0x300000bd0:27632 extent [0-1048575], client 4c063186/4c063186, server 49c97eab, cksum_type 1
LustreError: 21267:0:(osc_request.c:2449:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880260488f40 x1814444697701504/t0(0) o3->lustre-OST0003-osc-ffff8803240a2e98@0@lo:6/4 lens 488/440 e 0 to 0 dl 1730390612 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 job:'cmp.0' uid:0 gid:0
LustreError: lustre-OST0003: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x2000013a1:0x5190:0x0] object 0x300000bd0:27632 extent [0-1048575], client returned csum 4c063186 (type 1), server csum 49c97eab (type 1)

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31b81d063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix i2c_core virtio_blk serio_raw libata floppy [last unloaded: libcfs]
CPU: 13 PID: 12272 Comm: ptlrpcd_06_01 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880052a949d0 ti: ffff88009a330000 task.ti: ffff88009a330000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff88009a333948 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff88028be05070 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff88009a333988 R08: ffffffff818152c0 R09: ffffffff810baff0
R10: 0000000000000000 R11: 000000000000000f R12: ffff880331d5bf00
R13: ffffffff81dbbf10 R14: ffff880331d5bf44 R15: 0000000000690000
FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000001dad94000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa06007e0>] _debug_req+0x80/0x8b0 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff81068309>] ? kvm_sched_clock_read+0x9/0x20
[<ffffffff81035ec9>] ? sched_clock+0x9/0x10
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff8110a245>] ? __raw_callee_save___pv_queued_spin_unlock_slowpath+0x15/0x24
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffff817e3277>] ? _raw_spin_unlock_irqrestore+0x17/0x30
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa0620b00>] ? ptlrpcd_add_req+0x160/0x3f0 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa08c44f6>] brw_interpret+0xd66/0xe20 [osc]
[<ffffffffa05f0d98>] ptlrpc_check_set+0x3d8/0x2220 [ptlrpc]
[<ffffffffa06219e4>] ptlrpcd+0xaa4/0xb80 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa0620f40>] ? ptlrpcd_ctl_init+0x1b0/0x1b0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

LustreError: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x2000013a1:0x4cfa:0x0] object 0x240000bd0:26586 extent [0-1048575]: client csum 30ec5402, server csum 30ec5401
LustreError: lustre-OST0000-osc-ffff8801bd47e678: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 0@lo inode [0x2000013a1:0x4cfa:0x0] object 0x240000bd0:26586 extent [0-1048575], original client csum 30ec5402 (type 20), server csum 30ec5401 (type 20), client csum now 30ec5401
Lustre: *** cfs_fail_loc=408, val=0***
Lustre: Skipped 1 previous similar message
LustreError: lustre-OST0000-osc-ffff8801bd47e678: BAD READ CHECKSUM: from 0@lo inode [0x2000013a1:0x4cfa:0x0] object 0x240000bd0:26586 extent [3145728-4194303], client 53195492/53195492, server 30ec5401, cksum_type 20
LustreError: Skipped 1 previous similar message
LustreError: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x2000013a1:0x4cfa:0x0] object 0x240000bd0:26586 extent [3145728-4194303], client returned csum 53195492 (type 20), server csum 30ec5401 (type 20)
LustreError: Skipped 1 previous similar message

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31be54063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy libata i2c_core [last unloaded: libcfs]
CPU: 15 PID: 21706 Comm: ptlrpcd_07_00 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b71349d0 ti: ffff880092b60000 task.ti: ffff880092b60000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff880092b63950 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff8802c7f86ef0 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff880092b63990 R08: ffffffff818157c0 R09: ffffffff810baff0
R10: 0000000000000000 R11: 000000000000000f R12: ffff880331ddbf00
R13: ffffffff81dbbf10 R14: ffff880331ddbf44 R15: 0000000000790000
FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000002e6e36000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa0685e20>] _debug_req+0x80/0x890 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff81068309>] ? kvm_sched_clock_read+0x9/0x20
[<ffffffff81035ec9>] ? sched_clock+0x9/0x10
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffff817e3277>] ? _raw_spin_unlock_irqrestore+0x17/0x30
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa06a5770>] ? ptlrpcd_add_req+0x170/0x400 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa09182da>] brw_interpret+0xcca/0xd80 [osc]
[<ffffffffa0676b58>] ptlrpc_check_set+0x428/0x2180 [ptlrpc]
[<ffffffffa06a69e4>] ptlrpcd+0xa94/0xb70 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa06a5f50>] ? ptlrpcd_partners+0x3a0/0x3a0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: *** cfs_fail_loc=409, val=0***
LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200001b71:0x6125:0x0] object 0x280000bd0:32000 extent [0-1048575]: client csum 30ec5402, server csum 30ec5401
LustreError: 132-0: lustre-OST0000-osc-ffff8802afb95d28: BAD WRITE CHECKSUM: changed on the client after we checksummed it - likely false positive due to mmap IO (bug 11742): from 0@lo inode [0x200001b71:0x6125:0x0] object 0x280000bd0:32000 extent [0-1048575], original client csum 30ec5402 (type 20), server csum 30ec5401 (type 20), client csum now 30ec5401
LustreError: 21706:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff88025eb6d4c0 x1772600018895936/t21474874798(21474874798) o4->lustre-OST0000-osc-ffff8802afb95d28@0@lo:6/4 lens 488/448 e 0 to 0 dl 1690484429 ref 2 fl Interpret:RMQU/200/0 rc 0/0 uid:0 gid:0 job:'directio.0'
LustreError: 133-1: lustre-OST0000-osc-ffff8802afb95d28: BAD READ CHECKSUM: from 0@lo inode [0x200001b71:0x6125:0x0] object 0x280000bd0:32000 extent [0-1048575], client 53195492/53195492, server 30ec5401, cksum_type 20
LustreError: 132-0: lustre-OST0000: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b71:0x6125:0x0] object 0x280000bd0:32000 extent [0-1048575], client returned csum 53195492 (type 20), server csum 30ec5401 (type 20)

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31bf60063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs]
CPU: 2 PID: 31923 Comm: ptlrpcd_01_01 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802c8a35c40 ti: ffff8801b7708000 task.ti: ffff8801b7708000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff8801b770b950 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff880323f4a770 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff8801b770b990 R08: ffffffff81813c40 R09: ffffffff810baff0
R10: 0000000000000000 R11: 000000000000000f R12: ffff880331a9bf00
R13: ffffffff81dbbf10 R14: ffff880331a9bf44 R15: 0000000000110000
FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000002c655c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa0613de0>] _debug_req+0x80/0x890 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff81068309>] ? kvm_sched_clock_read+0x9/0x20
[<ffffffff81035ec9>] ? sched_clock+0x9/0x10
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffff817e3277>] ? _raw_spin_unlock_irqrestore+0x17/0x30
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa0633720>] ? ptlrpcd_add_req+0x170/0x400 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa08f82da>] brw_interpret+0xcca/0xd80 [osc]
[<ffffffffa0604b58>] ptlrpc_check_set+0x428/0x2180 [ptlrpc]
[<ffffffffa0634994>] ptlrpcd+0xa94/0xb70 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa0633f00>] ? ptlrpcd_partners+0x3a0/0x3a0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0
Lustre: *** cfs_fail_loc=408, val=0***
LustreError: 133-1: lustre-OST0002-osc-ffff8802d3192548: BAD READ CHECKSUM: from 0@lo inode [0x200001b71:0x5ff7:0x0] object 0x2c0000401:31379 extent [0-1048575], client 794146d1/794146d1, server 80a1047b, cksum_type 1
LustreError: 31921:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880320ea20c0 x1771640881916544/t0(0) o3->lustre-OST0002-osc-ffff8802d3192548@0@lo:6/4 lens 488/440 e 0 to 0 dl 1689569701 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'cmp.0'
LustreError: 132-0: lustre-OST0002: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b71:0x5ff7:0x0] object 0x2c0000401:31379 extent [0-1048575], client returned csum 794146d1 (type 1), server csum 80a1047b (type 1)
Lustre: DEBUG MARKER: set checksum type to adler, rc = 0
Lustre: *** cfs_fail_loc=408, val=0***
LustreError: 133-1: lustre-OST0002-osc-ffff8802d3192548: BAD READ CHECKSUM: from 0@lo inode [0x200001b71:0x5ff7:0x0] object 0x2c0000401:31379 extent [0-1048575], client f2ad1591/f2ad1591, server b64e1551, cksum_type 2
LustreError: 31920:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8801abc8d4c0 x1771640881918528/t0(0) o3->lustre-OST0002-osc-ffff8802d3192548@0@lo:6/4 lens 488/440 e 0 to 0 dl 1689569702 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'cmp.0'
LustreError: 132-0: lustre-OST0002: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b71:0x5ff7:0x0] object 0x2c0000401:31379 extent [0-1048575], client returned csum f2ad1591 (type 2), server csum b64e1551 (type 2)
Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0
Lustre: *** cfs_fail_loc=408, val=0***
LustreError: 133-1: lustre-OST0003-osc-ffff8802d3192548: BAD READ CHECKSUM: from 0@lo inode [0x200001b71:0x5ff7:0x0] object 0x300000401:31344 extent [0-1048575], client 15504bf0/15504bf0, server d0b82c03, cksum_type 4
LustreError: 31923:0:(osc_request.c:2409:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880323f48d40 x1771640881921920/t0(0) o3->lustre-OST0003-osc-ffff8802d3192548@0@lo:6/4 lens 488/440 e 0 to 0 dl 1689569704 ref 2 fl Interpret:RMQU/200/0 rc 1048576/1048576 uid:0 gid:0 job:'cmp.0'
LustreError: 132-0: lustre-OST0003: BAD READ CHECKSUM: should have changed on the client or in transit: from 0@lo inode [0x200001b71:0x5ff7:0x0] object 0x300000401:31344 extent [0-1048575], client returned csum 15504bf0 (type 4), server csum d0b82c03 (type 4)

BUG: unable to handle kernel paging request at ffffffff81dbbf10
IP: [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
PGD 1c14067 PUD 1c15063 PMD 31bf32063 PTE 8000000001dbb062
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: dm_flakey dm_mod lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks serio_raw virtio_blk floppy ata_piix i2c_core libata [last unloaded: libcfs]
CPU: 13 PID: 3502 Comm: ptlrpcd_06_01 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8802b4ec3760 ti: ffff88031af88000 task.ti: ffff88031af88000
RIP: 0010:[<ffffffff8110a9b2>] [<ffffffff8110a9b2>] __pv_queued_spin_lock_slowpath+0x1f2/0x3d0
RSP: 0018:ffff88031af8b950 EFLAGS: 00010282
RAX: 0000000000008000 RBX: ffff8802001c61f0 RCX: 0000000000000001
RDX: 0000000000000010 RSI: 0000000000000000 RDI: ffffffff81dbbf10
RBP: ffff88031af8b990 R08: ffffffff818152c0 R09: ffffffff810baff0
R10: 0000000000000000 R11: 000000000000000f R12: ffff880331d5bf00
R13: ffffffff81dbbf10 R14: ffff880331d5bf44 R15: 0000000000690000
FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: ffffffff81dbbf10 CR3: 00000001a1bc4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff8141192d>] do_raw_spin_lock+0x6d/0xa0
[<ffffffff817e320e>] _raw_spin_lock+0x1e/0x20
[<ffffffffa05f03b0>] _debug_req+0x80/0x890 [ptlrpc]
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff810682e3>] ? kvm_clock_read+0x33/0x40
[<ffffffff81068309>] ? kvm_sched_clock_read+0x9/0x20
[<ffffffff81035ec9>] ? sched_clock+0x9/0x10
[<ffffffff8110ae0a>] ? __pv_queued_spin_unlock_slowpath+0x5a/0xa0
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffff817e3277>] ? _raw_spin_unlock_irqrestore+0x17/0x30
[<ffffffff810c8631>] ? __wake_up_common_lock+0x91/0xc0
[<ffffffff810c7f10>] ? sched_feat_set+0xf0/0xf0
[<ffffffff810c8673>] ? __wake_up+0x13/0x20
[<ffffffffa0610a00>] ? ptlrpcd_add_req+0x170/0x400 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa09091ab>] brw_interpret+0xcdb/0xdc0 [osc]
[<ffffffffa05e1b28>] ptlrpc_check_set+0x408/0x2280 [ptlrpc]
[<ffffffffa0611c74>] ptlrpcd+0xa94/0xb70 [ptlrpc]
[<ffffffff810baff0>] ? abort_exclusive_wait+0xa0/0xa0
[<ffffffffa06111e0>] ? ptlrpcd_partners+0x3a0/0x3a0 [ptlrpc]
[<ffffffff810ba114>] kthread+0xe4/0xf0
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140
[<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21
[<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140

Lustre: DEBUG MARKER: set checksum type to crc32, rc = 0
LustreError: 168-f: lustre-OST0000: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x2c0000bd0:6502 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11
LustreError: 132-0: lustre-OST0000-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x2c0000bd0:6502 extent [0-1048575], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12
LustreError: 168-f: lustre-OST0002: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x340000bd0:6565 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11
LustreError: Skipped 8 previous similar messages
LustreError: 132-0: lustre-OST0000-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x2c0000bd0:6502 extent [0-1048575], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12
LustreError: Skipped 8 previous similar messages
LustreError: 168-f: lustre-OST0002: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x340000bd0:6565 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11
LustreError: Skipped 7 previous similar messages
LustreError: 132-0: lustre-OST0002-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x340000bd0:6565 extent [0-1048575], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12
LustreError: Skipped 7 previous similar messages
LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11
LustreError: Skipped 14 previous similar messages
LustreError: 132-0: lustre-OST0001-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [0-1048575], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12
LustreError: Skipped 14 previous similar messages
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 89 previous similar messages
LustreError: 3503:0:(osc_request.c:2403:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802e9abc7c0 x1763319366623936/t12884941735(12884941735) o4->lustre-OST0001-osc-ffff88028bcf8008@0@lo:6/4 lens 488/448 e 0 to 0 dl 1681633815 ref 2 fl Interpret:RMQU/200/0 rc 0/0 uid:0 gid:0 job:'ptlrpcd_07_00.0'
LustreError: 3503:0:(osc_request.c:2403:osc_brw_redo_request()) Skipped 41 previous similar messages
LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11
LustreError: Skipped 15 previous similar messages
LustreError: 132-0: lustre-OST0001-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [0-1048575], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12
LustreError: Skipped 15 previous similar messages
LustreError: 168-f: lustre-OST0001: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [0-1048575]: client csum b2f1b12, server csum b2f1b11
LustreError: Skipped 23 previous similar messages
LustreError: 132-0: lustre-OST0001-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [0-1048575], original client csum b2f1b12 (type 1), server csum b2f1b11 (type 1), client csum now b2f1b12
LustreError: Skipped 23 previous similar messages
LustreError: 3503:0:(osc_request.c:2533:brw_interpret()) lustre-OST0001-osc-ffff88028bcf8008: too many resent retries for object: 12884904912:6853, rc = -11.
LustreError: 3492:0:(osc_request.c:2533:brw_interpret()) lustre-OST0002-osc-ffff88028bcf8008: too many resent retries for object: 13958646736:6565, rc = -11.
LustreError: 3492:0:(osc_request.c:2533:brw_interpret()) Skipped 4 previous similar messages
Lustre: DEBUG MARKER: set checksum type to adler, rc = 0
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 111 previous similar messages
LustreError: 3503:0:(osc_request.c:2403:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff8802cdcd86c0 x1763319366654272/t21474875653(21474875653) o4->lustre-OST0000-osc-ffff88028bcf8008@0@lo:6/4 lens 488/448 e 0 to 0 dl 1681633848 ref 2 fl Interpret:RMQU/200/0 rc 0/0 uid:0 gid:0 job:'ptlrpcd_07_00.0'
LustreError: 3503:0:(osc_request.c:2403:osc_brw_redo_request()) Skipped 47 previous similar messages
LustreError: 168-f: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x380000bd0:6533 extent [1048576-2097151]: client csum 19eeae62, server csum 19eeae61
LustreError: Skipped 71 previous similar messages
LustreError: 132-0: lustre-OST0002-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x340000bd0:6565 extent [1048576-2097151], original client csum 19eeae62 (type 2), server csum 19eeae61 (type 2), client csum now 19eeae62
LustreError: Skipped 71 previous similar messages
LustreError: 3496:0:(osc_request.c:2533:brw_interpret()) lustre-OST0002-osc-ffff88028bcf8008: too many resent retries for object: 13958646736:6565, rc = -11.
LustreError: 3496:0:(osc_request.c:2533:brw_interpret()) Skipped 2 previous similar messages
Lustre: DEBUG MARKER: set checksum type to crc32c, rc = 0
Lustre: *** cfs_fail_loc=409, val=0***
Lustre: Skipped 255 previous similar messages
LustreError: 3501:0:(osc_request.c:2403:osc_brw_redo_request()) @@@ redo for recoverable error -11 req@ffff880224ddc7c0 x1763319366708736/t12884941579(12884941579) o4->lustre-OST0003-osc-ffff88028bcf8008@0@lo:6/4 lens 488/448 e 0 to 0 dl 1681633914 ref 2 fl Interpret:RMQU/200/0 rc 0/0 uid:0 gid:0 job:'ptlrpcd_06_00.0'
LustreError: 3501:0:(osc_request.c:2403:osc_brw_redo_request()) Skipped 119 previous similar messages
LustreError: 3501:0:(osc_request.c:2533:brw_interpret()) lustre-OST0002-osc-ffff88028bcf8008: too many resent retries for object: 13958646736:6565, rc = -11.
LustreError: 3501:0:(osc_request.c:2533:brw_interpret()) Skipped 7 previous similar messages
LustreError: 168-f: lustre-OST0003: BAD WRITE CHECKSUM: from 12345-0@lo inode [0x200002b11:0x3fd1:0x0] object 0x380000bd0:6533 extent [1048576-2097151]: client csum b5ea7f3c, server csum b5ea7f3b
LustreError: Skipped 118 previous similar messages
LustreError: 132-0: lustre-OST0001-osc-ffff88028bcf8008: BAD WRITE CHECKSUM: changed in transit before arrival at OST: from 0@lo inode [0x200002b11:0x3fd1:0x0] object 0x300000bd0:6853 extent [1048576-2097151], original client csum b5ea7f3c (type 4), server csum b5ea7f3b (type 4), client csum now b5ea7f3c
LustreError: Skipped 116 previous similar messages
Lustre: DEBUG MARKER: set checksum type to t10ip512, rc = 0
LustreError: 3501:0:(osc_request.c:2533:brw_interpret()) lustre-OST0000-osc-ffff88028bcf8008: too many resent retries for object: 11811163088:6502, rc = -11.
LustreError: 3501:0:(osc_request.c:2533:brw_interpret()) Skipped 7 previous similar messages
Lustre: DEBUG MARKER: set checksum type to t10ip4K, rc = 0
LustreError: 3502:0:(osc_request.c:2533:brw_interpret()) lustre-OST0002-osc-ffff88028bcf8008: too many resent retries for object: 13958646736:6565, rc = -11.
LustreError: 3502:0:(osc_request.c:2533:brw_interpret()) Skipped 7 previous similar messages
Lustre: DEBUG MARKER: set checksum type to t10crc512, rc = 0