BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31f26a067 PUD 31a983067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm ata_piix drm_panel_orientation_quirks serio_raw floppy libata i2c_core virtio_blk [last unloaded: libcfs]
CPU: 1 PID: 1011 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031af124f0 ti: ffff88031a2c0000 task.ti: ffff88031a2c0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a2c3e50 EFLAGS: 00010246
RAX: ffff88031a2c3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a53a98 RDI: 0000000000000063
RBP: ffff88031a2c3e50 R08: 0000000000000000 R09: 00000000000008f1
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f243de5c740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ac78000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.6M 521M 0 126 0 296K
lustre-mdt1 55.6M 520M 0 240 0 2.67M
lustre-mdt1 55.7M 520M 0 168 0 2.33M
lustre-mdt1 55.7M 520M 0 204 0 2.39M
lustre-mdt1 56.1M 520M 0 21 0 113K
lustre-mdt1 55.4M 521M 0 116 0 1.01M
lustre-mdt1 55.4M 521M 0 1 0 1.58K
lustre-mdt1 55.3M 521M 0 181 0 1.94M
lustre-mdt1 53.9M 522M 0 86 0 924K
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31bff3067 PUD 31d4ca067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk floppy serio_raw i2c_core [last unloaded: libcfs]
CPU: 14 PID: 997 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8803203b1280 ti: ffff880319f18000 task.ti: ffff880319f18000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880319f1be50 EFLAGS: 00010246
RAX: ffff880319f1bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff880319f1be50 R08: 0000000000000000 R09: 0000000000000937
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fb49bdbf740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031d0c4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.8M 521M 0 149 0 1.99M
lustre-mdt1 54.8M 521M 0 79 0 1.93M
lustre-mdt1 54.9M 521M 0 75 0 1.96M
lustre-mdt1 54.9M 521M 0 1 0 1.71K
lustre-mdt1 55.4M 521M 0 108 0 2.29M
lustre-mdt1 55.4M 521M 0 5 0 173K
lustre-mdt1 55.1M 521M 0 95 0 1.45M
lustre-mdt1 55.1M 521M 0 1 0 1.82K
lustre-mdt1 53.9M 522M 0 180 0 2.92M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31d98a067 PUD 31bb97067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks ata_piix serio_raw floppy i2c_core virtio_blk libata [last unloaded: libcfs]
CPU: 1 PID: 1023 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ee149d0 ti: ffff880320378000 task.ti: ffff880320378000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88032037be50 EFLAGS: 00010246
RAX: ffff88032037bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a53a98 RDI: 0000000000000063
RBP: ffff88032037be50 R08: 0000000000000000 R09: 0000000000000944
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f4dc772b740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031bb54000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 56.2M 520M 0 78 0 1.05M
lustre-mdt1 55.2M 521M 0 31 0 776K
lustre-mdt1 56.0M 520M 0 105 0 1.48M
lustre-mdt1 55.5M 520M 0 81 0 1.74M
lustre-mdt1 55.4M 521M 0 116 0 1.86M
lustre-mdt1 55.4M 521M 0 4 0 139K
lustre-mdt1 55.4M 521M 0 76 0 1.60M
lustre-mdt1 55.4M 521M 0 106 0 1.80M
lustre-mdt1 54.9M 521M 0 81 0 1.19M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bbad6067 PUD ba56b067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk i2c_core serio_raw floppy [last unloaded: libcfs]
CPU: 6 PID: 1000 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031a1fb760 ti: ffff8800baf1c000 task.ti: ffff8800baf1c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800baf1fe50 EFLAGS: 00010246
RAX: ffff8800baf1ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800baf1fe50 R08: 0000000000000000 R09: 0000000000000926
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007ff7768e4740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000baa68000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.3M 522M 0 77 0 1.24M
lustre-mdt1 54.4M 522M 0 80 0 1.69M
lustre-mdt1 54.7M 521M 0 78 0 2.26M
lustre-mdt1 54.7M 521M 0 0 0 701
lustre-mdt1 54.7M 521M 0 98 0 2.37M
lustre-mdt1 54.7M 521M 0 0 0 965
lustre-mdt1 54.7M 521M 0 1 0 1.53K
lustre-mdt1 55.9M 520M 0 92 0 2.68M
lustre-mdt1 54.8M 521M 0 74 0 992K
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31d29a067 PUD 31d29b067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 6 PID: 984 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031eef3760 ti: ffff8800ba1e0000 task.ti: ffff8800ba1e0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba1e3e50 EFLAGS: 00010246
RAX: ffff8800ba1e3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba1e3e50 R08: 0000000000000000 R09: 0000000000000936
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f6691bbc740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b98a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 55.4M 521M 0 50 0 165K
lustre-mdt1 54.5M 522M 0 100 0 1.22M
lustre-mdt1 54.5M 522M 0 0 0 755
lustre-mdt1 54.5M 522M 0 0 0 875
lustre-mdt1 54.8M 521M 0 80 0 1.32M
lustre-mdt1 54.8M 521M 0 0 0 731
lustre-mdt1 54.8M 521M 0 70 0 1.79M
lustre-mdt1 54.6M 521M 0 31 0 205K
lustre-mdt1 54.3M 522M 0 112 0 1.51M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 319645067 PUD 31bba5067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy virtio_blk libata i2c_core [last unloaded: libcfs]
CPU: 1 PID: 990 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b95dc9d0 ti: ffff88031e900000 task.ti: ffff88031e900000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031e903e50 EFLAGS: 00010246
RAX: ffff88031e903fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a53a98 RDI: 0000000000000063
RBP: ffff88031e903e50 R08: 0000000000000000 R09: 0000000000002155
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007efde828c740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000031a6da000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 53.9M 522M 0 93 0 1.46M
lustre-mdt1 53.9M 522M 0 0 0 907
lustre-mdt1 53.9M 522M 0 179 0 3.70M
lustre-mdt1 53.5M 522M 0 115 0 1.19M
lustre-mdt1 53.5M 522M 0 0 0 963
lustre-mdt1 53.8M 522M 0 107 0 2.51M
lustre-mdt1 53.8M 522M 0 0 0 944
lustre-mdt1 53.7M 522M 0 113 0 2.84M
lustre-mdt1 52.1M 524M 0 154 0 1.60M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b3dd067 PUD 31b259067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy i2c_core virtio_blk serio_raw libata [last unloaded: libcfs]
CPU: 2 PID: 988 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800bb0f1280 ti: ffff88031cdd0000 task.ti: ffff88031cdd0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031cdd3e50 EFLAGS: 00010246
RAX: ffff88031cdd3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031cdd3e50 R08: 0000000000000000 R09: 00000000000008d9
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f224809f740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031aaa6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 56.4M 520M 0 123 0 2.53M
lustre-mdt1 56.4M 520M 0 1 0 1.45K
lustre-mdt1 56.8M 519M 0 139 0 2.62M
lustre-mdt1 56.8M 519M 0 46 0 707K
lustre-mdt1 56.1M 520M 0 40 0 91.9K
lustre-mdt1 56.1M 520M 0 54 0 1.53M
lustre-mdt1 55.6M 520M 0 189 0 2.35M
lustre-mdt1 55.6M 520M 0 0 0 674
lustre-mdt1 55.9M 520M 0 78 0 1.40M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31eafe067 PUD 31eafd067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ata_piix ttm drm libata drm_panel_orientation_quirks virtio_blk floppy serio_raw i2c_core [last unloaded: libcfs]
CPU: 2 PID: 995 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e3ab760 ti: ffff88031d90c000 task.ti: ffff88031d90c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031d90fe50 EFLAGS: 00010246
RAX: ffff88031d90ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031d90fe50 R08: 0000000000000000 R09: 0000000000002087
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007ff5b1078740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031c6ca000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x2c0000405 to 0x2c0000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 55.2M 521M 0 104 0 766K
lustre-mdt1 55.1M 521M 0 125 0 1.36M
lustre-mdt1 54.4M 522M 0 137 0 1.67M
lustre-mdt1 54.4M 522M 0 0 0 827
lustre-mdt1 54.7M 521M 0 206 0 2.18M
lustre-mdt1 54.7M 521M 0 52 0 858K
lustre-mdt1 54.8M 521M 0 157 0 1.45M
lustre-mdt1 54.8M 521M 0 0 0 797
lustre-mdt1 54.8M 521M 0 137 0 2.12M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a923067 PUD 31bb58067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw i2c_core virtio_blk libata floppy [last unloaded: libcfs]
CPU: 0 PID: 991 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b1949d0 ti: ffff88031cf54000 task.ti: ffff88031cf54000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031cf57e50 EFLAGS: 00010246
RAX: ffff88031cf57fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031cf57e50 R08: 0000000000000000 R09: 0000000000002117
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f5d46ec9740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031dc62000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x2c0000405 to 0x2c0000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 53.5M 522M 0 120 0 942K
lustre-mdt1 53.5M 522M 0 0 0 850
lustre-mdt1 53.5M 522M 0 1 0 1.62K
lustre-mdt1 53.5M 522M 0 157 0 2.05M
lustre-mdt1 53.9M 522M 0 74 0 149K
lustre-mdt1 53.9M 522M 0 0 0 841
lustre-mdt1 53.9M 522M 0 107 0 1.74M
lustre-mdt1 53.6M 522M 0 70 0 144K
lustre-mdt1 53.6M 522M 0 0 0 840
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31981b067 PUD 31f14e067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks serio_raw ata_piix floppy virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 15 PID: 994 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800baafb760 ti: ffff8800ba878000 task.ti: ffff8800ba878000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba87be50 EFLAGS: 00010246
RAX: ffff8800ba87bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba87be50 R08: 0000000000000000 R09: 0000000000003fcb
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fe4400a0740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000003194bc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 57.2M 519M 0 128 0 3.62M
lustre-mdt1 55.8M 520M 0 111 0 2.24M
lustre-mdt1 55.5M 520M 0 113 0 1.19M
lustre-mdt1 56.7M 519M 0 95 0 2.75M
lustre-mdt1 56.7M 519M 0 209 0 3.12M
lustre-mdt1 56.2M 520M 0 98 0 1.78M
lustre-mdt1 56.0M 520M 0 180 0 3.63M
lustre-mdt1 57.6M 518M 0 118 0 3.66M
lustre-mdt1 57.7M 518M 0 206 0 2.57M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb9ed067 PUD bb9eb067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs]
CPU: 5 PID: 997 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ac00010 ti: ffff8800b8d7c000 task.ti: ffff8800b8d7c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b8d7fe50 EFLAGS: 00010246
RAX: ffff8800b8d7ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800b8d7fe50 R08: 0000000000000000 R09: 00000000000037a6
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f5476254740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000bb96c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 58.4M 518M 0 127 0 1.21M
lustre-mdt1 58.9M 517M 0 142 0 2.79M
lustre-mdt1 58.5M 518M 0 125 0 3.26M
lustre-mdt1 56.3M 520M 0 252 0 3.23M
lustre-mdt1 55.3M 521M 0 109 0 1.13M
lustre-mdt1 56.3M 520M 0 136 0 3.48M
lustre-mdt1 56.3M 520M 0 0 0 979
lustre-mdt1 56.8M 519M 0 181 0 3.76M
lustre-mdt1 56.3M 520M 0 78 0 1.29M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31ae86067 PUD 31ae87067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk libata serio_raw i2c_core floppy [last unloaded: libcfs]
CPU: 0 PID: 958 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880319cf1280 ti: ffff88031cf14000 task.ti: ffff88031cf14000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031cf17e50 EFLAGS: 00010246
RAX: ffff88031cf17fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a13a98 RDI: 0000000000000063
RBP: ffff88031cf17e50 R08: 0000000000000000 R09: 00000000000008c3
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fb1d07ea740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ef2c000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.8M 521M 0 524 0 7.59M
lustre-mdt1 54.7M 521M 0 149 0 4.25M
lustre-mdt1 54.7M 521M 0 231 0 2.74M
lustre-mdt1 53.6M 522M 0 282 0 5.02M
lustre-mdt1 54.3M 522M 0 153 0 3.02M
lustre-mdt1 54.8M 521M 0 229 0 3.89M
lustre-mdt1 53.1M 523M 0 192 0 2.28M
lustre-mdt1 53.5M 523M 0 127 0 2.78M
lustre-mdt1 52.9M 523M 0 110 0 1.79M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31cf92067 PUD 319d8b067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm serio_raw drm_panel_orientation_quirks floppy virtio_blk libata i2c_core [last unloaded: libcfs]
CPU: 3 PID: 972 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b9550010 ti: ffff88031e4dc000 task.ti: ffff88031e4dc000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031e4dfe50 EFLAGS: 00010246
RAX: ffff88031e4dffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331ad3a98 RDI: 0000000000000063
RBP: ffff88031e4dfe50 R08: 0000000000000000 R09: 0000000000002086
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f1d46abd740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031a4b6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.4M 525M 0 748 0 7.19M
lustre-mdt1 52.5M 523M 0 308 0 3.29M
lustre-mdt1 53.5M 523M 0 419 0 4.11M
lustre-mdt1 53.1M 523M 0 84 0 316K
lustre-mdt1 52.9M 523M 0 507 0 4.92M
lustre-mdt1 49.2M 527M 0 479 0 4.31M
lustre-mdt1 50.3M 526M 0 413 0 4.43M
lustre-mdt1 51.8M 524M 0 322 0 3.36M
lustre-mdt1 51.7M 524M 0 308 0 2.86M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31adcb067 PUD 31e544067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix i2c_core floppy virtio_blk serio_raw libata [last unloaded: libcfs]
CPU: 14 PID: 1022 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e9eb760 ti: ffff88031d318000 task.ti: ffff88031d318000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031d31be50 EFLAGS: 00010246
RAX: ffff88031d31bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff88031d31be50 R08: 0000000000000000 R09: 0000000000000a9d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f92a7038740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319f58000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802d25a4138: disconnect after 20s idle
Lustre: Skipped 3 previous similar messages
bash (26258): drop_caches: 1
bash (28957): drop_caches: 1
bash (30696): drop_caches: 1
bash (786): drop_caches: 1
bash (2763): drop_caches: 1
bash (4767): drop_caches: 1
bash (7483): drop_caches: 1
bash (9313): drop_caches: 1
bash (11977): drop_caches: 1
bash (13782): drop_caches: 1
bash (16370): drop_caches: 1
bash (18095): drop_caches: 1
bash (20816): drop_caches: 1
bash (22547): drop_caches: 1
bash (25173): drop_caches: 1
bash (26878): drop_caches: 1
bash (29381): drop_caches: 1
bash (31002): drop_caches: 1
bash (1368): drop_caches: 1
bash (3623): drop_caches: 1
bash (6524): drop_caches: 1
bash (8312): drop_caches: 1
bash (11112): drop_caches: 1
bash (12973): drop_caches: 1
bash (15627): drop_caches: 1
bash (17296): drop_caches: 1
bash (19924): drop_caches: 1
bash (21843): drop_caches: 1
bash (24597): drop_caches: 1
bash (26525): drop_caches: 1
bash (28903): drop_caches: 1
bash (30802): drop_caches: 1
Lustre: 2273:0:(llog_cat.c:971:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0001: catlog [0x1:0x2:0x0] crosses index zero
bash (955): drop_caches: 1
bash (3128): drop_caches: 1
bash (6273): drop_caches: 1
bash (8137): drop_caches: 1
bash (10733): drop_caches: 1
bash (12515): drop_caches: 1
bash (14967): drop_caches: 1
bash (16809): drop_caches: 1
bash (19414): drop_caches: 1
bash (21306): drop_caches: 1
bash (24033): drop_caches: 1
bash (26001): drop_caches: 1
bash (28668): drop_caches: 1
bash (30484): drop_caches: 1
bash (690): drop_caches: 1
bash (2694): drop_caches: 1
bash (4728): drop_caches: 1
bash (7548): drop_caches: 1
bash (9445): drop_caches: 1
bash (12161): drop_caches: 1
bash (13952): drop_caches: 1
bash (16226): drop_caches: 1
bash (18018): drop_caches: 1
bash (20596): drop_caches: 1
bash (22364): drop_caches: 1
bash (24988): drop_caches: 1
bash (26716): drop_caches: 1
bash (29293): drop_caches: 1
bash (31049): drop_caches: 1
bash (1591): drop_caches: 1
bash (3727): drop_caches: 1
bash (6782): drop_caches: 1
bash (8865): drop_caches: 1
bash (11745): drop_caches: 1
bash (13541): drop_caches: 1
bash (16108): drop_caches: 1
bash (17719): drop_caches: 1
bash (20067): drop_caches: 1
bash (21703): drop_caches: 1
bash (24119): drop_caches: 1
bash (26018): drop_caches: 1
bash (28076): drop_caches: 1
bash (29785): drop_caches: 1
bash (32011): drop_caches: 1
bash (1384): drop_caches: 1
bash (4151): drop_caches: 1
bash (6025): drop_caches: 1
bash (8354): drop_caches: 1
bash (10138): drop_caches: 1
bash (12366): drop_caches: 1
bash (14032): drop_caches: 1
bash (16756): drop_caches: 1
bash (18514): drop_caches: 1
bash (20758): drop_caches: 1
bash (22378): drop_caches: 1
bash (24671): drop_caches: 1
bash (26593): drop_caches: 1
bash (29126): drop_caches: 1
bash (30987): drop_caches: 1
bash (32637): drop_caches: 1
bash (3069): drop_caches: 1
bash (4943): drop_caches: 1
bash (7513): drop_caches: 1
bash (9187): drop_caches: 1
bash (11262): drop_caches: 1
bash (12951): drop_caches: 1
bash (15310): drop_caches: 1
bash (16906): drop_caches: 1
bash (19177): drop_caches: 1
bash (20821): drop_caches: 1
bash (23200): drop_caches: 1
bash (24930): drop_caches: 1
bash (27731): drop_caches: 1
bash (29264): drop_caches: 1
bash (31553): drop_caches: 1
bash (838): drop_caches: 1
bash (3956): drop_caches: 1
bash (5865): drop_caches: 1
bash (8429): drop_caches: 1
bash (9987): drop_caches: 1
bash (12483): drop_caches: 1
bash (14213): drop_caches: 1
bash (16557): drop_caches: 1
bash (18073): drop_caches: 1
bash (20173): drop_caches: 1
bash (21767): drop_caches: 1
bash (24173): drop_caches: 1
bash (25767): drop_caches: 1
bash (27923): drop_caches: 1
bash (29499): drop_caches: 1
bash (31863): drop_caches: 1
bash (1053): drop_caches: 1
bash (4009): drop_caches: 1
bash (5912): drop_caches: 1
bash (8280): drop_caches: 1
bash (9910): drop_caches: 1
bash (12316): drop_caches: 1
bash (14070): drop_caches: 1
bash (16493): drop_caches: 1
bash (18045): drop_caches: 1
bash (19628): drop_caches: 1
bash (22070): drop_caches: 1
bash (23655): drop_caches: 1
bash (25955): drop_caches: 1
bash (27699): drop_caches: 1
bash (29845): drop_caches: 1
bash (31582): drop_caches: 1
bash (1739): drop_caches: 1
bash (3578): drop_caches: 1
bash (6125): drop_caches: 1
bash (7784): drop_caches: 1
bash (10324): drop_caches: 1
bash (12066): drop_caches: 1
bash (14400): drop_caches: 1
bash (16015): drop_caches: 1
bash (18359): drop_caches: 1
bash (20107): drop_caches: 1
bash (22535): drop_caches: 1
bash (24160): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 20.2M 556M 0 0 0 981
lustre-mdt1 20.2M 556M 0 0 0 964
lustre-mdt1 20.2M 556M 0 0 0 982
lustre-mdt1 20.2M 556M 0 0 0 990
lustre-mdt1 20.2M 556M 0 0 0 983
lustre-mdt1 20.2M 556M 0 0 0 988
lustre-mdt1 20.2M 556M 0 0 0 984
lustre-mdt1 20.2M 556M 0 0 0 967
lustre-mdt1 20.2M 556M 0 0 0 982
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b911067 PUD 320411067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 9 PID: 996 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031bb424f0 ti: ffff8800bacd0000 task.ti: ffff8800bacd0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bacd3e50 EFLAGS: 00010246
RAX: ffff8800bacd3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800bacd3e50 R08: 0000000000000000 R09: 0000000000000a6d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fc4da813740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031e538000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff880208510008: disconnect after 24s idle
Lustre: Skipped 3 previous similar messages
bash (3766): drop_caches: 1
bash (4201): drop_caches: 1
bash (4647): drop_caches: 1
bash (5102): drop_caches: 1
bash (5586): drop_caches: 1
bash (6004): drop_caches: 1
bash (6466): drop_caches: 1
bash (6950): drop_caches: 1
bash (7365): drop_caches: 1
bash (7804): drop_caches: 1
bash (8326): drop_caches: 1
bash (8835): drop_caches: 1
bash (9204): drop_caches: 1
bash (9658): drop_caches: 1
bash (10248): drop_caches: 1
bash (10852): drop_caches: 1
bash (11271): drop_caches: 1
bash (11705): drop_caches: 1
bash (12257): drop_caches: 1
bash (12671): drop_caches: 1
bash (13114): drop_caches: 1
bash (13546): drop_caches: 1
bash (13982): drop_caches: 1
bash (14415): drop_caches: 1
bash (14995): drop_caches: 1
bash (15394): drop_caches: 1
bash (15837): drop_caches: 1
bash (16309): drop_caches: 1
bash (16784): drop_caches: 1
bash (17243): drop_caches: 1
bash (17681): drop_caches: 1
bash (18094): drop_caches: 1
bash (18940): drop_caches: 1
bash (20542): drop_caches: 1
bash (22119): drop_caches: 1
bash (23713): drop_caches: 1
bash (25251): drop_caches: 1
bash (27148): drop_caches: 1
bash (29091): drop_caches: 1
bash (30957): drop_caches: 1
bash (32667): drop_caches: 1
bash (1864): drop_caches: 1
bash (3489): drop_caches: 1
bash (5172): drop_caches: 1
bash (6724): drop_caches: 1
bash (8296): drop_caches: 1
bash (9891): drop_caches: 1
bash (11573): drop_caches: 1
bash (13290): drop_caches: 1
bash (14891): drop_caches: 1
bash (16517): drop_caches: 1
bash (18191): drop_caches: 1
bash (19886): drop_caches: 1
bash (21407): drop_caches: 1
bash (22944): drop_caches: 1
bash (24514): drop_caches: 1
bash (26100): drop_caches: 1
bash (28130): drop_caches: 1
bash (29894): drop_caches: 1
bash (31599): drop_caches: 1
bash (889): drop_caches: 1
bash (2607): drop_caches: 1
bash (4228): drop_caches: 1
bash (5870): drop_caches: 1
bash (7408): drop_caches: 1
bash (8970): drop_caches: 1
bash (10665): drop_caches: 1
bash (12312): drop_caches: 1
bash (14012): drop_caches: 1
bash (15702): drop_caches: 1
bash (17254): drop_caches: 1
bash (18831): drop_caches: 1
bash (20419): drop_caches: 1
bash (22132): drop_caches: 1
bash (23798): drop_caches: 1
bash (25531): drop_caches: 1
bash (27305): drop_caches: 1
bash (29289): drop_caches: 1
bash (30885): drop_caches: 1
bash (32530): drop_caches: 1
bash (1672): drop_caches: 1
bash (3289): drop_caches: 1
bash (4718): drop_caches: 1
bash (6279): drop_caches: 1
bash (7763): drop_caches: 1
bash (9408): drop_caches: 1
bash (10906): drop_caches: 1
bash (12408): drop_caches: 1
bash (14055): drop_caches: 1
bash (15698): drop_caches: 1
bash (17204): drop_caches: 1
bash (18750): drop_caches: 1
bash (20309): drop_caches: 1
bash (21911): drop_caches: 1
bash (23506): drop_caches: 1
bash (25049): drop_caches: 1
bash (26784): drop_caches: 1
bash (28671): drop_caches: 1
bash (30428): drop_caches: 1
bash (32025): drop_caches: 1
bash (1156): drop_caches: 1
bash (2739): drop_caches: 1
bash (4473): drop_caches: 1
bash (6047): drop_caches: 1
bash (7727): drop_caches: 1
bash (9344): drop_caches: 1
bash (10845): drop_caches: 1
bash (12393): drop_caches: 1
bash (13988): drop_caches: 1
bash (15538): drop_caches: 1
bash (17071): drop_caches: 1
bash (18605): drop_caches: 1
bash (20099): drop_caches: 1
bash (21598): drop_caches: 1
bash (23066): drop_caches: 1
bash (24599): drop_caches: 1
bash (25994): drop_caches: 1
bash (27708): drop_caches: 1
bash (29445): drop_caches: 1
bash (31263): drop_caches: 1
bash (425): drop_caches: 1
bash (1931): drop_caches: 1
bash (3587): drop_caches: 1
bash (5061): drop_caches: 1
bash (6465): drop_caches: 1
bash (7776): drop_caches: 1
bash (9288): drop_caches: 1
bash (10826): drop_caches: 1
bash (12375): drop_caches: 1
bash (13825): drop_caches: 1
bash (15477): drop_caches: 1
bash (17034): drop_caches: 1
bash (18446): drop_caches: 1
bash (19868): drop_caches: 1
bash (21407): drop_caches: 1
bash (22875): drop_caches: 1
bash (24385): drop_caches: 1
bash (25903): drop_caches: 1
bash (27708): drop_caches: 1
bash (29248): drop_caches: 1
bash (30874): drop_caches: 1
bash (32464): drop_caches: 1
bash (1453): drop_caches: 1
bash (2899): drop_caches: 1
bash (4492): drop_caches: 1
bash (6110): drop_caches: 1
bash (7543): drop_caches: 1
bash (9070): drop_caches: 1
bash (10532): drop_caches: 1
bash (12123): drop_caches: 1
bash (13572): drop_caches: 1
bash (15034): drop_caches: 1
bash (16476): drop_caches: 1
bash (18004): drop_caches: 1
bash (19593): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 43.3M 533M 0 0 0 949
lustre-mdt1 43.3M 533M 0 0 0 982
lustre-mdt1 43.3M 533M 0 0 0 958
lustre-mdt1 43.3M 533M 0 1 0 1.84K
lustre-mdt1 43.3M 533M 0 0 0 904
lustre-mdt1 43.3M 533M 0 0 0 961
lustre-mdt1 43.3M 533M 0 0 0 970
lustre-mdt1 43.3M 533M 0 0 0 970
lustre-mdt1 43.3M 533M 0 0 0 958
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb096067 PUD bbb5b067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core floppy libata [last unloaded: libcfs]
CPU: 3 PID: 985 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8803192624f0 ti: ffff880320348000 task.ti: ffff880320348000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88032034be50 EFLAGS: 00010246
RAX: ffff88032034bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331ad3a98 RDI: 0000000000000063
RBP: ffff88032034be50 R08: 0000000000000000 R09: 000000000000090b
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f6686d30740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000bb272000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 48.5M 527M 0 318 0 3.88M
lustre-mdt1 49.5M 527M 0 193 0 2.37M
lustre-mdt1 48.7M 527M 0 154 0 1.98M
lustre-mdt1 48.8M 527M 0 120 0 2.09M
lustre-mdt1 48.7M 527M 0 158 0 2.36M
lustre-mdt1 47.9M 528M 0 286 0 3.34M
lustre-mdt1 48.1M 528M 0 194 0 2.55M
lustre-mdt1 48.1M 528M 0 177 0 3.15M
lustre-mdt1 48.8M 527M 0 26 0 39.6K
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31d0f9067 PUD 3188d2067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks floppy ata_piix serio_raw i2c_core libata virtio_blk [last unloaded: libcfs]
CPU: 2 PID: 1000 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b8c5c40 ti: ffff88031b380000 task.ti: ffff88031b380000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b383e50 EFLAGS: 00010246
RAX: ffff88031b383fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a93a98 RDI: 0000000000000063
RBP: ffff88031b383e50 R08: 0000000000000000 R09: 000000000000acea
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fdf650e9740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b900000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x300000405 to 0x300000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 53.2M 523M 0 252 0 3.49M
lustre-mdt1 54.2M 522M 0 138 0 2.28M
lustre-mdt1 54.5M 521M 0 522 0 5.54M
lustre-mdt1 55.8M 520M 0 585 0 7.49M
lustre-mdt1 56.5M 519M 0 377 0 4.87M
lustre-mdt1 57.4M 519M 0 179 0 2.17M
lustre-mdt1 58.4M 518M 0 181 0 3.28M
lustre-mdt1 57.4M 519M 0 263 0 2.42M
lustre-mdt1 54.4M 522M 0 394 0 4.75M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3197cc067 PUD 320a05067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core floppy libata [last unloaded: libcfs]
CPU: 13 PID: 1015 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b3fa4f0 ti: ffff88031f1ac000 task.ti: ffff88031f1ac000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031f1afe50 EFLAGS: 00010246
RAX: ffff88031f1affd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d53a98 RDI: 0000000000000063
RBP: ffff88031f1afe50 R08: 0000000000000000 R09: 000000000000660e
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f9008902740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031baac000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.5M 522M 0 124 0 2.14M
lustre-mdt1 53.8M 522M 0 371 0 5.65M
lustre-mdt1 51.8M 524M 0 240 0 3.03M
lustre-mdt1 51.4M 525M 0 219 0 4.27M
lustre-mdt1 51.3M 525M 0 272 0 3.68M
lustre-mdt1 51.3M 525M 0 12 0 800K
lustre-mdt1 51.1M 525M 0 210 0 3.56M
lustre-mdt1 51.1M 525M 0 95 0 1.41M
lustre-mdt1 51.1M 525M 0 12 0 246K
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb935067 PUD b97c3067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 13 PID: 963 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e5724f0 ti: ffff8800b9168000 task.ti: ffff8800b9168000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b916be50 EFLAGS: 00010246
RAX: ffff8800b916bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d53a98 RDI: 0000000000000063
RBP: ffff8800b916be50 R08: 0000000000000000 R09: 00000000000008eb
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fc2032b1740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000bb9c0000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 53.3M 523M 0 608 0 6.87M
lustre-mdt1 53.3M 523M 0 574 0 6.22M
lustre-mdt1 54.2M 522M 0 457 0 5.20M
lustre-mdt1 54.0M 522M 0 561 0 5.66M
lustre-mdt1 55.1M 521M 0 258 0 3.32M
lustre-mdt1 55.4M 521M 0 249 0 3.31M
lustre-mdt1 51.1M 525M 0 675 0 5.81M
lustre-mdt1 51.2M 525M 0 246 0 2.40M
lustre-mdt1 51.2M 525M 0 425 0 4.65M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb971067 PUD bb91c067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy [last unloaded: libcfs]
CPU: 5 PID: 995 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031af2c9d0 ti: ffff88031b07c000 task.ti: ffff88031b07c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b07fe50 EFLAGS: 00010246
RAX: ffff88031b07ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b53a98 RDI: 0000000000000063
RBP: ffff88031b07fe50 R08: 0000000000000000 R09: 00000000000020ff
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f616de1f740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000ba2ce000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 50.8M 525M 0 544 0 7.31M
lustre-mdt1 53.0M 523M 0 353 0 3.54M
lustre-mdt1 54.1M 522M 0 462 0 5.27M
lustre-mdt1 52.8M 523M 0 448 0 4.55M
lustre-mdt1 52.3M 524M 0 575 0 6.05M
lustre-mdt1 54.3M 522M 0 545 0 6.82M
lustre-mdt1 55.9M 520M 0 554 0 6.04M
lustre-mdt1 55.1M 521M 0 531 0 5.74M
lustre-mdt1 53.9M 522M 0 512 0 5.61M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b8af067 PUD 318de0067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 7 PID: 1022 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031babb760 ti: ffff88031a8d4000 task.ti: ffff88031a8d4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a8d7e50 EFLAGS: 00010246
RAX: ffff88031a8d7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331bd3a98 RDI: 0000000000000063
RBP: ffff88031a8d7e50 R08: 0000000000000000 R09: 0000000000000ab7
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007ff1acab0740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000320336000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802d3444a88: disconnect after 24s idle
Lustre: Skipped 7 previous similar messages
bash (27400): drop_caches: 1
bash (29628): drop_caches: 1
bash (31289): drop_caches: 1
bash (1132): drop_caches: 1
bash (2837): drop_caches: 1
bash (4847): drop_caches: 1
bash (6593): drop_caches: 1
bash (9251): drop_caches: 1
bash (11063): drop_caches: 1
bash (13632): drop_caches: 1
bash (15431): drop_caches: 1
bash (17718): drop_caches: 1
bash (19584): drop_caches: 1
bash (22175): drop_caches: 1
bash (24318): drop_caches: 1
bash (27164): drop_caches: 1
bash (28771): drop_caches: 1
bash (31409): drop_caches: 1
bash (918): drop_caches: 1
bash (3335): drop_caches: 1
bash (4988): drop_caches: 1
bash (7420): drop_caches: 1
bash (9180): drop_caches: 1
bash (11455): drop_caches: 1
bash (13096): drop_caches: 1
bash (15266): drop_caches: 1
bash (16848): drop_caches: 1
bash (19138): drop_caches: 1
bash (20809): drop_caches: 1
bash (23159): drop_caches: 1
bash (24941): drop_caches: 1
Lustre: 22772:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (27286): drop_caches: 1
bash (28956): drop_caches: 1
bash (31290): drop_caches: 1
bash (592): drop_caches: 1
bash (2953): drop_caches: 1
bash (4892): drop_caches: 1
bash (7481): drop_caches: 1
bash (9203): drop_caches: 1
bash (11759): drop_caches: 1
bash (13363): drop_caches: 1
bash (15613): drop_caches: 1
bash (17277): drop_caches: 1
bash (19212): drop_caches: 1
bash (21707): drop_caches: 1
bash (23437): drop_caches: 1
bash (26417): drop_caches: 1
bash (28240): drop_caches: 1
bash (30780): drop_caches: 1
bash (32268): drop_caches: 1
bash (2175): drop_caches: 1
bash (3801): drop_caches: 1
bash (6300): drop_caches: 1
bash (8126): drop_caches: 1
bash (10581): drop_caches: 1
bash (12313): drop_caches: 1
bash (14861): drop_caches: 1
bash (16644): drop_caches: 1
bash (19314): drop_caches: 1
bash (21004): drop_caches: 1
bash (23688): drop_caches: 1
bash (25732): drop_caches: 1
bash (27966): drop_caches: 1
bash (29650): drop_caches: 1
bash (32099): drop_caches: 1
bash (1326): drop_caches: 1
bash (3735): drop_caches: 1
bash (5603): drop_caches: 1
bash (8017): drop_caches: 1
bash (9951): drop_caches: 1
bash (12674): drop_caches: 1
bash (14431): drop_caches: 1
bash (16682): drop_caches: 1
bash (18413): drop_caches: 1
bash (20605): drop_caches: 1
bash (22395): drop_caches: 1
bash (25347): drop_caches: 1
bash (27423): drop_caches: 1
bash (29932): drop_caches: 1
bash (31650): drop_caches: 1
bash (1538): drop_caches: 1
bash (3049): drop_caches: 1
bash (5288): drop_caches: 1
bash (6843): drop_caches: 1
bash (9617): drop_caches: 1
bash (11290): drop_caches: 1
bash (12866): drop_caches: 1
bash (15080): drop_caches: 1
bash (16756): drop_caches: 1
bash (18993): drop_caches: 1
bash (20517): drop_caches: 1
bash (22871): drop_caches: 1
bash (24833): drop_caches: 1
bash (27698): drop_caches: 1
bash (29385): drop_caches: 1
Lustre: 22772:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (31706): drop_caches: 1
bash (1116): drop_caches: 1
bash (3244): drop_caches: 1
bash (4972): drop_caches: 1
bash (7261): drop_caches: 1
bash (8936): drop_caches: 1
bash (11361): drop_caches: 1
bash (13190): drop_caches: 1
bash (15726): drop_caches: 1
bash (17330): drop_caches: 1
bash (19629): drop_caches: 1
bash (21482): drop_caches: 1
bash (24239): drop_caches: 1
bash (26269): drop_caches: 1
bash (28549): drop_caches: 1
bash (30158): drop_caches: 1
bash (368): drop_caches: 1
bash (2195): drop_caches: 1
bash (4722): drop_caches: 1
bash (6271): drop_caches: 1
bash (8619): drop_caches: 1
bash (10335): drop_caches: 1
bash (12830): drop_caches: 1
bash (14433): drop_caches: 1
bash (16881): drop_caches: 1
bash (18522): drop_caches: 1
bash (20849): drop_caches: 1
bash (22587): drop_caches: 1
bash (25556): drop_caches: 1
bash (27319): drop_caches: 1
bash (29592): drop_caches: 1
bash (31204): drop_caches: 1
bash (32726): drop_caches: 1
bash (2500): drop_caches: 1
bash (3989): drop_caches: 1
bash (5983): drop_caches: 1
bash (7569): drop_caches: 1
bash (9665): drop_caches: 1
bash (11434): drop_caches: 1
hrtimer: interrupt took 4424310 ns
bash (13607): drop_caches: 1
bash (15186): drop_caches: 1
bash (17519): drop_caches: 1
bash (19087): drop_caches: 1
bash (21294): drop_caches: 1
bash (23095): drop_caches: 1
bash (26020): drop_caches: 1
bash (28342): drop_caches: 1
bash (31013): drop_caches: 1
bash (419): drop_caches: 1
bash (3240): drop_caches: 1
bash (5124): drop_caches: 1
bash (7485): drop_caches: 1
bash (9077): drop_caches: 1
bash (11425): drop_caches: 1
bash (12881): drop_caches: 1
bash (15249): drop_caches: 1
bash (17088): drop_caches: 1
bash (19180): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 45.5M 530M 0 0 0 978
lustre-mdt1 45.5M 530M 0 0 0 978
lustre-mdt1 45.5M 530M 0 0 0 978
lustre-mdt1 45.5M 530M 0 0 0 980
lustre-mdt1 43.3M 533M 0 0 0 979
bash (20858): drop_caches: 1
lustre-mdt1 43.3M 533M 0 0 0 986
lustre-mdt1 43.3M 533M 0 0 0 991
lustre-mdt1 45.2M 531M 0 379 0 4.34M
lustre-mdt1 49.2M 527M 0 185 0 2.11M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b353067 PUD 31a2da067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm ata_piix floppy virtio_blk drm_panel_orientation_quirks serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 1 PID: 1014 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031934c9d0 ti: ffff88031f2b0000 task.ti: ffff88031f2b0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031f2b3e50 EFLAGS: 00010246
RAX: ffff88031f2b3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031f2b3e50 R08: 0000000000000000 R09: 000000000000675d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f9528f9c740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031f2ac000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 55.1M 521M 0 381 0 6.15M
lustre-mdt1 52.3M 524M 0 650 0 9.85M
lustre-mdt1 52.0M 524M 0 250 0 3.40M
lustre-mdt1 52.7M 523M 0 504 0 8.87M
lustre-mdt1 53.5M 523M 0 530 0 9.11M
lustre-mdt1 55.0M 521M 0 205 0 3.43M
lustre-mdt1 52.9M 523M 0 690 0 9.96M
lustre-mdt1 53.5M 522M 0 112 0 2.52M
lustre-mdt1 52.5M 524M 0 180 0 2.92M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31c6c8067 PUD 31ba18067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 4 PID: 999 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88032047dc40 ti: ffff88031a02c000 task.ti: ffff88031a02c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a02fe50 EFLAGS: 00010246
RAX: ffff88031a02ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b13a98 RDI: 0000000000000063
RBP: ffff88031a02fe50 R08: 0000000000000000 R09: 00000000000066b5
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f3389937740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000031ba20000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

bash (6518): drop_caches: 1
Lustre: ll_ost00_029: service thread pid 19169 was inactive for 40.096 seconds. The thread might be hung, or it might only be slow and will resume later. Dumping the stack trace for debugging purposes:
Pid: 19169, comm: ll_ost00_029 3.10.0-7.9-debug #2 SMP Tue Feb 1 18:17:58 EST 2022
Call Trace:
[<0>] cv_wait_common+0xb0/0x150 [spl]
[<0>] __cv_wait_io+0x18/0x20 [spl]
[<0>] txg_wait_synced_impl+0xdd/0x130 [zfs]
[<0>] txg_wait_synced+0x10/0x50 [zfs]
[<0>] dmu_tx_wait+0x26d/0x3a0 [zfs]
[<0>] dmu_tx_assign+0x91/0x480 [zfs]
[<0>] osd_trans_start+0x1a7/0x480 [osd_zfs]
[<0>] ofd_precreate_objects+0xa51/0x2380 [ofd]
[<0>] ofd_create_hdl+0x424/0x2400 [ofd]
[<0>] tgt_request_handle+0x74e/0x1a60 [ptlrpc]
[<0>] ptlrpc_server_handle_request+0x257/0xcd0 [ptlrpc]
[<0>] ptlrpc_main+0xc61/0x1640 [ptlrpc]
[<0>] kthread+0xe4/0xf0
[<0>] ret_from_fork_nospec_begin+0x7/0x21
[<0>] 0xfffffffffffffffe
bash (20357): drop_caches: 1
Lustre: ll_ost00_029: service thread pid 19169 completed after 48.294s. This likely indicates the system was overloaded (too many service threads, or not enough hardware resources).
bash (25426): drop_caches: 1
bash (29775): drop_caches: 1
bash (13562): drop_caches: 1
bash (19977): drop_caches: 1
bash (23500): drop_caches: 1
bash (32202): drop_caches: 1
bash (11581): drop_caches: 1
bash (16115): drop_caches: 1
bash (24752): drop_caches: 1
bash (32065): drop_caches: 1
bash (4085): drop_caches: 1
bash (13766): drop_caches: 1
bash (22080): drop_caches: 1
bash (450): drop_caches: 1
bash (14513): drop_caches: 1
bash (26651): drop_caches: 1
bash (31454): drop_caches: 1
bash (4541): drop_caches: 1
bash (8142): drop_caches: 1
bash (12644): drop_caches: 1
bash (17159): drop_caches: 1
bash (21345): drop_caches: 1
bash (3010): drop_caches: 1
bash (15716): drop_caches: 1
bash (28978): drop_caches: 1
bash (2645): drop_caches: 1
bash (6840): drop_caches: 1
bash (10683): drop_caches: 1
bash (14783): drop_caches: 1
bash (18553): drop_caches: 1
bash (22865): drop_caches: 1
bash (24843): drop_caches: 1
bash (26835): drop_caches: 1
bash (29047): drop_caches: 1
bash (31045): drop_caches: 1
bash (934): drop_caches: 1
bash (3142): drop_caches: 1
bash (5270): drop_caches: 1
bash (7271): drop_caches: 1
bash (9371): drop_caches: 1
bash (11513): drop_caches: 1
bash (13566): drop_caches: 1
bash (15531): drop_caches: 1
bash (17649): drop_caches: 1
bash (19675): drop_caches: 1
bash (21836): drop_caches: 1
bash (23733): drop_caches: 1
bash (25782): drop_caches: 1
bash (27770): drop_caches: 1
bash (29708): drop_caches: 1
bash (31804): drop_caches: 1
bash (1817): drop_caches: 1
bash (4103): drop_caches: 1
bash (6175): drop_caches: 1
bash (8300): drop_caches: 1
bash (10498): drop_caches: 1
bash (12453): drop_caches: 1
bash (14625): drop_caches: 1
bash (16679): drop_caches: 1
bash (18535): drop_caches: 1
bash (20525): drop_caches: 1
bash (22675): drop_caches: 1
bash (24789): drop_caches: 1
bash (26735): drop_caches: 1
bash (28768): drop_caches: 1
bash (30911): drop_caches: 1
bash (893): drop_caches: 1
bash (3319): drop_caches: 1
bash (5478): drop_caches: 1
bash (7450): drop_caches: 1
bash (9691): drop_caches: 1
bash (11753): drop_caches: 1
bash (13901): drop_caches: 1
bash (15925): drop_caches: 1
bash (18002): drop_caches: 1
bash (20186): drop_caches: 1
bash (22213): drop_caches: 1
bash (24187): drop_caches: 1
bash (26297): drop_caches: 1
bash (28268): drop_caches: 1
bash (30242): drop_caches: 1
bash (32721): drop_caches: 1
bash (2678): drop_caches: 1
bash (4909): drop_caches: 1
bash (6949): drop_caches: 1
bash (8966): drop_caches: 1
bash (10764): drop_caches: 1
bash (12525): drop_caches: 1
bash (14439): drop_caches: 1
bash (16471): drop_caches: 1
bash (18440): drop_caches: 1
bash (20309): drop_caches: 1
bash (22203): drop_caches: 1
bash (24183): drop_caches: 1
bash (26301): drop_caches: 1
bash (28269): drop_caches: 1
bash (30333): drop_caches: 1
bash (32511): drop_caches: 1
bash (2251): drop_caches: 1
bash (4493): drop_caches: 1
bash (6463): drop_caches: 1
bash (8516): drop_caches: 1
bash (10595): drop_caches: 1
bash (12454): drop_caches: 1
bash (14376): drop_caches: 1
bash (16220): drop_caches: 1
bash (18011): drop_caches: 1
bash (19999): drop_caches: 1
bash (21888): drop_caches: 1
bash (24092): drop_caches: 1
bash (26036): drop_caches: 1
bash (27976): drop_caches: 1
bash (29919): drop_caches: 1
bash (32058): drop_caches: 1
bash (2027): drop_caches: 1
bash (4056): drop_caches: 1
bash (6139): drop_caches: 1
bash (8049): drop_caches: 1
bash (9962): drop_caches: 1
bash (11988): drop_caches: 1
bash (14018): drop_caches: 1
bash (16013): drop_caches: 1
bash (18092): drop_caches: 1
bash (19996): drop_caches: 1
bash (22151): drop_caches: 1
bash (24034): drop_caches: 1
bash (25952): drop_caches: 1
bash (27847): drop_caches: 1
bash (29774): drop_caches: 1
bash (31661): drop_caches: 1
bash (1618): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 27.2M 549M 0 0 0 992
lustre-mdt1 27.2M 549M 0 0 0 993
lustre-mdt1 24.5M 551M 0 0 0 983
bash (3740): drop_caches: 1
lustre-mdt1 24.5M 551M 0 0 0 997
lustre-mdt1 26.2M 550M 0 210 0 3.05M
lustre-mdt1 29.8M 546M 0 284 0 3.88M
lustre-mdt1 31.7M 544M 0 223 0 4.18M
lustre-mdt1 34.5M 541M 0 260 0 3.28M
lustre-mdt1 38.3M 538M 0 314 0 5.07M
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a9a7067 PUD 318ce5067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw i2c_core libata virtio_blk [last unloaded: libcfs]
CPU: 3 PID: 1004 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800bb02a4f0 ti: ffff8800b948c000 task.ti: ffff8800b948c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b948fe50 EFLAGS: 00010246
RAX: ffff8800b948ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331ad3a98 RDI: 0000000000000063
RBP: ffff8800b948fe50 R08: 0000000000000000 R09: 0000000000000a8a
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f1a46a6d740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031e910000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802c145b7e8: disconnect after 21s idle
Lustre: Skipped 11 previous similar messages
bash (20288): drop_caches: 1
bash (20750): drop_caches: 1
bash (21256): drop_caches: 1
bash (21672): drop_caches: 1
bash (22146): drop_caches: 1
bash (22628): drop_caches: 1
bash (23066): drop_caches: 1
bash (23582): drop_caches: 1
bash (24033): drop_caches: 1
bash (24550): drop_caches: 1
bash (25004): drop_caches: 1
bash (25538): drop_caches: 1
bash (25958): drop_caches: 1
bash (26397): drop_caches: 1
bash (26902): drop_caches: 1
bash (27345): drop_caches: 1
bash (27936): drop_caches: 1
bash (28442): drop_caches: 1
bash (29000): drop_caches: 1
bash (29492): drop_caches: 1
bash (29917): drop_caches: 1
bash (30551): drop_caches: 1
bash (31067): drop_caches: 1
bash (31525): drop_caches: 1
bash (32089): drop_caches: 1
bash (32704): drop_caches: 1
bash (842): drop_caches: 1
bash (1279): drop_caches: 1
bash (1932): drop_caches: 1
bash (2443): drop_caches: 1
bash (2865): drop_caches: 1
bash (3389): drop_caches: 1
bash (4237): drop_caches: 1
bash (5792): drop_caches: 1
bash (7443): drop_caches: 1
bash (9006): drop_caches: 1
bash (10494): drop_caches: 1
bash (12220): drop_caches: 1
bash (13816): drop_caches: 1
bash (15478): drop_caches: 1
bash (17084): drop_caches: 1
bash (18714): drop_caches: 1
bash (20301): drop_caches: 1
bash (21955): drop_caches: 1
bash (23610): drop_caches: 1
bash (25246): drop_caches: 1
bash (26861): drop_caches: 1
bash (28488): drop_caches: 1
bash (30058): drop_caches: 1
bash (31990): drop_caches: 1
bash (1326): drop_caches: 1
bash (3130): drop_caches: 1
bash (4770): drop_caches: 1
bash (6297): drop_caches: 1
bash (7797): drop_caches: 1
bash (9456): drop_caches: 1
bash (11071): drop_caches: 1
bash (12699): drop_caches: 1
bash (14476): drop_caches: 1
bash (15914): drop_caches: 1
bash (17445): drop_caches: 1
bash (19053): drop_caches: 1
bash (20645): drop_caches: 1
bash (22347): drop_caches: 1
bash (24025): drop_caches: 1
bash (25730): drop_caches: 1
bash (27361): drop_caches: 1
bash (29015): drop_caches: 1
bash (30770): drop_caches: 1
bash (32624): drop_caches: 1
bash (2000): drop_caches: 1
bash (3608): drop_caches: 1
bash (5259): drop_caches: 1
bash (6825): drop_caches: 1
bash (8350): drop_caches: 1
bash (9974): drop_caches: 1
bash (11590): drop_caches: 1
bash (13114): drop_caches: 1
bash (14607): drop_caches: 1
bash (16283): drop_caches: 1
bash (17855): drop_caches: 1
bash (19378): drop_caches: 1
bash (20894): drop_caches: 1
bash (22474): drop_caches: 1
bash (24078): drop_caches: 1
bash (25693): drop_caches: 1
bash (27291): drop_caches: 1
bash (28766): drop_caches: 1
bash (30469): drop_caches: 1
bash (32230): drop_caches: 1
bash (1757): drop_caches: 1
bash (3485): drop_caches: 1
bash (5081): drop_caches: 1
bash (6644): drop_caches: 1
bash (8355): drop_caches: 1
bash (10056): drop_caches: 1
bash (11641): drop_caches: 1
bash (13201): drop_caches: 1
bash (14771): drop_caches: 1
bash (16454): drop_caches: 1
bash (17975): drop_caches: 1
bash (19622): drop_caches: 1
bash (21227): drop_caches: 1
bash (22820): drop_caches: 1
bash (24415): drop_caches: 1
bash (25974): drop_caches: 1
bash (27426): drop_caches: 1
bash (29223): drop_caches: 1
bash (31125): drop_caches: 1
bash (405): drop_caches: 1
bash (2246): drop_caches: 1
bash (3778): drop_caches: 1
bash (5359): drop_caches: 1
bash (6962): drop_caches: 1
bash (8653): drop_caches: 1
bash (10230): drop_caches: 1
bash (11743): drop_caches: 1
bash (13369): drop_caches: 1
bash (15006): drop_caches: 1
bash (16492): drop_caches: 1
bash (18127): drop_caches: 1
bash (19698): drop_caches: 1
bash (21187): drop_caches: 1
bash (22700): drop_caches: 1
bash (24247): drop_caches: 1
bash (25875): drop_caches: 1
bash (27572): drop_caches: 1
bash (29117): drop_caches: 1
bash (30978): drop_caches: 1
bash (379): drop_caches: 1
bash (2180): drop_caches: 1
bash (3974): drop_caches: 1
bash (5482): drop_caches: 1
bash (7122): drop_caches: 1
bash (8654): drop_caches: 1
bash (10182): drop_caches: 1
bash (11798): drop_caches: 1
bash (13452): drop_caches: 1
bash (15100): drop_caches: 1
bash (16814): drop_caches: 1
bash (18355): drop_caches: 1
bash (19894): drop_caches: 1
bash (21475): drop_caches: 1
bash (23179): drop_caches: 1
bash (24819): drop_caches: 1
bash (26481): drop_caches: 1
bash (27927): drop_caches: 1
bash (29601): drop_caches: 1
bash (31497): drop_caches: 1
bash (976): drop_caches: 1
bash (2775): drop_caches: 1
bash (4372): drop_caches: 1
bash (5951): drop_caches: 1
bash (7432): drop_caches: 1
bash (9034): drop_caches: 1
bash (10527): drop_caches: 1
bash (12145): drop_caches: 1
bash (13680): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 42.9M 533M 0 0 0 961
lustre-mdt1 42.9M 533M 0 0 0 970
lustre-mdt1 42.9M 533M 0 0 0 979
lustre-mdt1 42.9M 533M 0 0 0 974
lustre-mdt1 42.9M 533M 0 0 0 982
lustre-mdt1 42.9M 533M 0 0 0 974
lustre-mdt1 42.9M 533M 0 0 0 985
lustre-mdt1 42.9M 533M 0 0 0 971
lustre-mdt1 42.9M 533M 0 0 0 951
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b1a8067 PUD 31b07a067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix virtio_blk serio_raw drm_panel_orientation_quirks floppy libata i2c_core [last unloaded: libcfs]
CPU: 10 PID: 990 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e6a1280 ti: ffff8803220e8000 task.ti: ffff8803220e8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803220ebe50 EFLAGS: 00010246
RAX: ffff8803220ebfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331c93a98 RDI: 0000000000000063
RBP: ffff8803220ebe50 R08: 0000000000000000 R09: 0000000000000a69
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f461f567740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031afca000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8800aba11bf8: disconnect after 21s idle
Lustre: Skipped 3 previous similar messages
bash (15824): drop_caches: 1
bash (16353): drop_caches: 1
bash (16851): drop_caches: 1
bash (17325): drop_caches: 1
bash (17763): drop_caches: 1
bash (18264): drop_caches: 1
bash (18769): drop_caches: 1
bash (19195): drop_caches: 1
bash (19655): drop_caches: 1
bash (20275): drop_caches: 1
bash (20769): drop_caches: 1
bash (21274): drop_caches: 1
bash (21738): drop_caches: 1
bash (22286): drop_caches: 1
bash (22747): drop_caches: 1
bash (23193): drop_caches: 1
bash (23623): drop_caches: 1
bash (24082): drop_caches: 1
bash (24496): drop_caches: 1
bash (24917): drop_caches: 1
bash (25429): drop_caches: 1
bash (25926): drop_caches: 1
bash (26362): drop_caches: 1
bash (26949): drop_caches: 1
bash (27448): drop_caches: 1
bash (27993): drop_caches: 1
bash (28448): drop_caches: 1
bash (28974): drop_caches: 1
bash (29642): drop_caches: 1
bash (30091): drop_caches: 1
bash (30625): drop_caches: 1
bash (31079): drop_caches: 1
bash (31978): drop_caches: 1
bash (1082): drop_caches: 1
bash (2633): drop_caches: 1
bash (4366): drop_caches: 1
bash (5908): drop_caches: 1
bash (7443): drop_caches: 1
bash (8903): drop_caches: 1
bash (10468): drop_caches: 1
bash (12036): drop_caches: 1
bash (13521): drop_caches: 1
bash (15160): drop_caches: 1
bash (16665): drop_caches: 1
bash (18172): drop_caches: 1
bash (19699): drop_caches: 1
bash (21268): drop_caches: 1
bash (22901): drop_caches: 1
bash (24584): drop_caches: 1
bash (26314): drop_caches: 1
bash (28399): drop_caches: 1
bash (30323): drop_caches: 1
bash (31882): drop_caches: 1
bash (867): drop_caches: 1
bash (2481): drop_caches: 1
bash (4026): drop_caches: 1
bash (5560): drop_caches: 1
bash (7194): drop_caches: 1
bash (8676): drop_caches: 1
bash (10247): drop_caches: 1
bash (11788): drop_caches: 1
bash (13447): drop_caches: 1
bash (14887): drop_caches: 1
bash (16453): drop_caches: 1
bash (18019): drop_caches: 1
bash (19578): drop_caches: 1
bash (21173): drop_caches: 1
bash (22743): drop_caches: 1
bash (24483): drop_caches: 1
bash (26083): drop_caches: 1
bash (28011): drop_caches: 1
bash (29955): drop_caches: 1
bash (31618): drop_caches: 1
bash (860): drop_caches: 1
bash (2560): drop_caches: 1
bash (4190): drop_caches: 1
bash (5779): drop_caches: 1
bash (7468): drop_caches: 1
bash (8967): drop_caches: 1
bash (10624): drop_caches: 1
bash (12260): drop_caches: 1
bash (13848): drop_caches: 1
bash (15485): drop_caches: 1
bash (17105): drop_caches: 1
bash (18594): drop_caches: 1
bash (20137): drop_caches: 1
bash (21509): drop_caches: 1
bash (23007): drop_caches: 1
bash (24562): drop_caches: 1
bash (26085): drop_caches: 1
bash (28035): drop_caches: 1
bash (29813): drop_caches: 1
bash (31526): drop_caches: 1
bash (690): drop_caches: 1
bash (2271): drop_caches: 1
bash (3775): drop_caches: 1
bash (5329): drop_caches: 1
bash (7034): drop_caches: 1
bash (8488): drop_caches: 1
bash (9993): drop_caches: 1
bash (11491): drop_caches: 1
bash (13094): drop_caches: 1
bash (14646): drop_caches: 1
bash (16271): drop_caches: 1
bash (17867): drop_caches: 1
bash (19348): drop_caches: 1
bash (20884): drop_caches: 1
bash (22376): drop_caches: 1
bash (23966): drop_caches: 1
bash (25431): drop_caches: 1
bash (27359): drop_caches: 1
bash (29064): drop_caches: 1
bash (30841): drop_caches: 1
bash (32699): drop_caches: 1
bash (1889): drop_caches: 1
bash (3420): drop_caches: 1
bash (5128): drop_caches: 1
bash (6669): drop_caches: 1
bash (8242): drop_caches: 1
bash (9766): drop_caches: 1
bash (11353): drop_caches: 1
bash (12849): drop_caches: 1
bash (14436): drop_caches: 1
bash (16016): drop_caches: 1
bash (17448): drop_caches: 1
bash (19109): drop_caches: 1
bash (20816): drop_caches: 1
bash (22322): drop_caches: 1
bash (23921): drop_caches: 1
bash (25386): drop_caches: 1
bash (27097): drop_caches: 1
bash (28979): drop_caches: 1
bash (30646): drop_caches: 1
bash (32294): drop_caches: 1
bash (1468): drop_caches: 1
bash (3132): drop_caches: 1
bash (4688): drop_caches: 1
bash (6266): drop_caches: 1
bash (7756): drop_caches: 1
bash (9373): drop_caches: 1
bash (10914): drop_caches: 1
bash (12481): drop_caches: 1
bash (14061): drop_caches: 1
bash (15804): drop_caches: 1
bash (17251): drop_caches: 1
bash (18752): drop_caches: 1
bash (20292): drop_caches: 1
bash (21760): drop_caches: 1
bash (23237): drop_caches: 1
bash (24839): drop_caches: 1
bash (26277): drop_caches: 1
bash (28157): drop_caches: 1
bash (29899): drop_caches: 1
bash (31530): drop_caches: 1
bash (647): drop_caches: 1
bash (2201): drop_caches: 1
bash (3685): drop_caches: 1
bash (5179): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 20.3M 556M 0 0 0 986
lustre-mdt1 20.3M 556M 0 0 0 988
lustre-mdt1 20.3M 556M 0 0 0 987
lustre-mdt1 20.3M 556M 0 0 0 983
lustre-mdt1 20.3M 556M 0 0 0 973
lustre-mdt1 20.3M 556M 0 0 0 984
lustre-mdt1 20.3M 556M 0 0 0 989
lustre-mdt1 20.3M 556M 0 0 0 985
bash (6690): drop_caches: 1
lustre-mdt1 20.3M 556M 0 1 0 1.88K
Lustre: OBD: obd_test_device_fini
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31aa8b067 PUD 31aa18067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix libata i2c_core serio_raw virtio_blk floppy [last unloaded: libcfs]
CPU: 13 PID: 986 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b81e9280 ti: ffff88031c690000 task.ti: ffff88031c690000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031c693e50 EFLAGS: 00010246
RAX: ffff88031c693fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d53a98 RDI: 0000000000000063
RBP: ffff88031c693e50 R08: 0000000000000000 R09: 0000000000006600
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f7b3bfd5740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b206000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.7M 521M 0 402 0 5.81M
lustre-mdt1 52.2M 524M 0 259 0 5.40M
lustre-mdt1 51.7M 524M 0 573 0 5.94M
lustre-mdt1 52.7M 523M 0 213 0 3.32M
lustre-mdt1 53.9M 522M 0 462 0 6.84M
lustre-mdt1 52.9M 523M 0 327 0 4.83M
lustre-mdt1 53.3M 523M 0 427 0 7.14M
lustre-mdt1 52.3M 524M 0 273 0 3.29M
lustre-mdt1 52.1M 524M 0 212 0 4.39M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb94d067 PUD ba670067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk libata i2c_core serio_raw [last unloaded: libcfs]
CPU: 14 PID: 993 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031fd60010 ti: ffff8800b95a8000 task.ti: ffff8800b95a8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b95abe50 EFLAGS: 00010246
RAX: ffff8800b95abfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff8800b95abe50 R08: 0000000000000000 R09: 0000000000003700
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fa06df7a740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000bba30000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.9M 524M 0 194 0 3.65M
lustre-mdt1 53.1M 523M 0 112 0 3.30M
lustre-mdt1 51.6M 524M 0 150 0 3.97M
lustre-mdt1 53.8M 522M 0 250 0 4.87M
lustre-mdt1 53.8M 522M 0 198 0 3.96M
lustre-mdt1 54.0M 522M 0 360 0 6.67M
lustre-mdt1 54.8M 521M 0 156 0 3.41M
lustre-mdt1 55.6M 520M 0 274 0 6.13M
lustre-mdt1 55.0M 521M 0 302 0 4.20M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 320464067 PUD 31df28067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk libata serio_raw i2c_core floppy [last unloaded: libcfs]
CPU: 2 PID: 974 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b1e5c40 ti: ffff8803200a0000 task.ti: ffff8803200a0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803200a3e50 EFLAGS: 00010246
RAX: ffff8803200a3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8803200a3e50 R08: 0000000000000000 R09: 0000000000009491
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f3231c93740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031fb70000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.7M 524M 0 308 0 4.66M
lustre-mdt1 50.1M 526M 0 285 0 5.78M
lustre-mdt1 50.3M 526M 0 200 0 3.50M
lustre-mdt1 51.3M 525M 0 182 0 3.17M
lustre-mdt1 52.1M 524M 0 121 0 3.12M
lustre-mdt1 49.8M 526M 0 261 0 3.91M
lustre-mdt1 49.3M 527M 0 237 0 4.46M
lustre-mdt1 49.3M 527M 0 87 0 3.13M
lustre-mdt1 50.2M 526M 0 166 0 1.52M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e435067 PUD 3198be067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw libata i2c_core floppy [last unloaded: libcfs]
CPU: 10 PID: 980 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b9521280 ti: ffff88031f2e8000 task.ti: ffff88031f2e8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031f2ebe50 EFLAGS: 00010246
RAX: ffff88031f2ebfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331c93a98 RDI: 0000000000000063
RBP: ffff88031f2ebe50 R08: 0000000000000000 R09: 00000000000008fb
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f8ba008d740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031f3be000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x2c0000405 to 0x2c0000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.4M 522M 0 233 0 2.91M
lustre-mdt1 52.9M 523M 0 296 0 3.61M
lustre-mdt1 52.4M 524M 0 190 0 3.34M
lustre-mdt1 52.4M 524M 0 0 0 986
lustre-mdt1 53.2M 523M 0 152 0 3.46M
lustre-mdt1 51.9M 524M 0 314 0 4.00M
lustre-mdt1 51.9M 524M 0 173 0 3.99M
lustre-mdt1 53.1M 523M 0 124 0 3.33M
lustre-mdt1 53.1M 523M 0 86 0 3.21M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b8ec067 PUD 31a73d067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 3 PID: 962 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031a5124f0 ti: ffff88031b1d8000 task.ti: ffff88031b1d8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b1dbe50 EFLAGS: 00010246
RAX: ffff88031b1dbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331ad3a98 RDI: 0000000000000063
RBP: ffff88031b1dbe50 R08: 0000000000000000 R09: 000000000000739c
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fa72dc8a740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000003194ae000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.2M 525M 0 152 0 2.82M
lustre-mdt1 52.0M 524M 0 190 0 4.61M
lustre-mdt1 56.1M 520M 0 272 0 4.14M
lustre-mdt1 54.7M 521M 0 300 0 2.89M
lustre-mdt1 53.7M 522M 0 421 0 5.68M
lustre-mdt1 52.4M 524M 0 172 0 1.83M
lustre-mdt1 53.3M 523M 0 228 0 4.76M
lustre-mdt1 53.9M 522M 0 493 0 4.83M
lustre-mdt1 53.9M 522M 0 115 0 1.93M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD b8c38067 PUD b8c3f067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 10 PID: 975 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e068010 ti: ffff8803202a8000 task.ti: ffff8803202a8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803202abe50 EFLAGS: 00010246
RAX: ffff8803202abfd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8803202abe50 R08: 0000000000000000 R09: 00000000000008dc
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f7152a50740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000bb92e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 50.7M 525M 0 284 0 3.80M
lustre-mdt1 50.7M 525M 0 118 0 2.17M
lustre-mdt1 50.8M 525M 0 288 0 2.32M
lustre-mdt1 52.3M 524M 0 250 0 2.96M
lustre-mdt1 50.7M 525M 0 640 0 7.19M
lustre-mdt1 50.1M 526M 0 176 0 2.04M
lustre-mdt1 51.1M 525M 0 261 0 3.27M
lustre-mdt1 51.5M 525M 0 331 0 4.22M
lustre-mdt1 52.3M 524M 0 269 0 3.35M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e0ec067 PUD 31a9a9067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy i2c_core serio_raw virtio_blk libata [last unloaded: libcfs]
CPU: 6 PID: 979 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e6a9280 ti: ffff8800baab0000 task.ti: ffff8800baab0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800baab3e50 EFLAGS: 00010246
RAX: ffff8800baab3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800baab3e50 R08: 0000000000000000 R09: 000000000000209d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f4acda15740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000031e800000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.3M 525M 0 388 0 3.33M
lustre-mdt1 51.1M 525M 0 109 0 733K
lustre-mdt1 49.2M 527M 0 715 0 6.98M
lustre-mdt1 49.6M 526M 0 278 0 2.98M
lustre-mdt1 47.8M 528M 0 421 0 5.10M
lustre-mdt1 49.3M 527M 0 376 0 3.79M
lustre-mdt1 51.2M 525M 0 389 0 4.38M
lustre-mdt1 50.9M 525M 0 417 0 4.00M
lustre-mdt1 49.8M 526M 0 382 0 4.50M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a65c067 PUD 31a437067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm serio_raw ata_piix virtio_blk drm_panel_orientation_quirks libata i2c_core floppy [last unloaded: libcfs]
CPU: 11 PID: 991 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031c5649d0 ti: ffff880319eec000 task.ti: ffff880319eec000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880319eefe50 EFLAGS: 00010246
RAX: ffff880319eeffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331cd3a98 RDI: 0000000000000063
RBP: ffff880319eefe50 R08: 0000000000000000 R09: 00000000000020a0
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f9ee1346740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b912000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 47.0M 529M 0 275 0 3.85M
lustre-mdt1 48.2M 528M 0 212 0 3.01M
lustre-mdt1 48.9M 527M 0 466 0 6.16M
lustre-mdt1 49.4M 527M 0 247 0 3.02M
lustre-mdt1 48.4M 528M 0 134 0 1.57M
lustre-mdt1 48.2M 528M 0 280 0 3.73M
lustre-mdt1 49.0M 527M 0 230 0 2.49M
lustre-mdt1 48.9M 527M 0 474 0 5.43M
lustre-mdt1 48.6M 527M 0 199 0 3.08M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b2f1067 PUD 31a23b067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk floppy serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 15 PID: 1005 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b085c40 ti: ffff8800bb3c8000 task.ti: ffff8800bb3c8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bb3cbe50 EFLAGS: 00010246
RAX: ffff8800bb3cbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331dd3a98 RDI: 0000000000000063
RBP: ffff8800bb3cbe50 R08: 0000000000000000 R09: 00000000000020f9
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f3a62110740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b07a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

hrtimer: interrupt took 72380380 ns
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 52.6M 523M 0 428 0 4.52M
lustre-mdt1 51.0M 525M 0 290 0 4.23M
lustre-mdt1 51.2M 525M 0 173 0 2.89M
lustre-mdt1 51.5M 524M 0 181 0 2.62M
lustre-mdt1 52.4M 524M 0 120 0 1.13M
lustre-mdt1 53.0M 523M 0 462 0 5.98M
lustre-mdt1 53.0M 523M 0 135 0 2.73M
lustre-mdt1 52.5M 523M 0 263 0 3.85M
lustre-mdt1 51.2M 525M 0 148 0 3.31M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3223b4067 PUD 3223b5067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata [last unloaded: obdecho]
CPU: 9 PID: 976 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ef224f0 ti: ffff8800ba774000 task.ti: ffff8800ba774000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba777e50 EFLAGS: 00010246
RAX: ffff8800ba777fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba777e50 R08: 0000000000000000 R09: 0000000000000f93
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fe8fc1c8740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000320314000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

LNet: *** cfs_fail_loc=e001, val=100***
LNet: 32663:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1829699201855616 with NID mismatch 0@lo accepted (same peer ffff88004d443508)
Lustre: DEBUG MARKER: sanity test_350: @@@@@@ FAIL: cp failed
LNet: *** cfs_fail_loc=e001, val=100***
LNet: Skipped 38 previous similar messages
LNet: 7561:0:(lib-ptl.c:160:lnet_try_match_md()) message from 12345-0@lo matched 1829699201861248 with NID mismatch 0@lo accepted (same peer ffff88004d443508)
LNet: 7561:0:(lib-ptl.c:160:lnet_try_match_md()) Skipped 38 previous similar messages
LustreError: 7694:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 29444:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 29444:0:(tgt_grant.c:452:tgt_grant_space_left()) Skipped 1 previous similar message
LustreError: 6893:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 6893:0:(tgt_grant.c:452:tgt_grant_space_left()) Skipped 2 previous similar messages
LustreError: 7631:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 7631:0:(tgt_grant.c:452:tgt_grant_space_left()) Skipped 1 previous similar message
LustreError: 6086:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 6086:0:(tgt_grant.c:452:tgt_grant_space_left()) Skipped 1 previous similar message
LustreError: 6893:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 6893:0:(tgt_grant.c:452:tgt_grant_space_left()) Skipped 1 previous similar message
LustreError: 7561:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0002: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff880320355d28 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
LustreError: 7561:0:(tgt_grant.c:452:tgt_grant_space_left()) Skipped 1 previous similar message
LustreError: 7561:0:(tgt_grant.c:452:tgt_grant_space_left()) lustre-MDT0001: cli 6f381594-9efd-4401-bffa-dc20f2462770/ffff8800a374ae98 left=0 < tot_grant=3407872 unstable=0 pending=0 dirty=0
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 23.4M 553M 0 0 0 984
lustre-mdt1 23.4M 553M 0 0 0 983
lustre-mdt1 23.4M 553M 0 0 0 981
lustre-mdt1 23.4M 553M 0 0 0 978
lustre-mdt1 23.4M 553M 0 0 0 984
lustre-mdt1 23.4M 553M 0 0 0 981
lustre-mdt1 23.4M 553M 0 0 0 985
lustre-mdt1 23.4M 553M 0 0 0 986
lustre-mdt1 23.4M 553M 0 0 0 986
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bbabb067 PUD bb9c6067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks i2c_core ata_piix serio_raw virtio_blk floppy libata [last unloaded: libcfs]
CPU: 0 PID: 1012 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e4e3760 ti: ffff8803209fc000 task.ti: ffff8803209fc000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803209ffe50 EFLAGS: 00010246
RAX: ffff8803209fffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8803209ffe50 R08: 0000000000000000 R09: 0000000000000a6b
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007ff6717ae740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000bb9da000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802dee0ae98: disconnect after 22s idle
Lustre: Skipped 12 previous similar messages
bash (32077): drop_caches: 1
bash (32526): drop_caches: 1
bash (454): drop_caches: 1
bash (981): drop_caches: 1
bash (1539): drop_caches: 1
bash (2030): drop_caches: 1
bash (2474): drop_caches: 1
bash (3085): drop_caches: 1
bash (3608): drop_caches: 1
bash (4174): drop_caches: 1
bash (4632): drop_caches: 1
bash (5183): drop_caches: 1
bash (5646): drop_caches: 1
bash (6066): drop_caches: 1
bash (6596): drop_caches: 1
bash (7032): drop_caches: 1
bash (7412): drop_caches: 1
bash (7876): drop_caches: 1
bash (8381): drop_caches: 1
bash (8963): drop_caches: 1
bash (9425): drop_caches: 1
bash (9854): drop_caches: 1
bash (10303): drop_caches: 1
bash (10812): drop_caches: 1
bash (11212): drop_caches: 1
bash (11689): drop_caches: 1
bash (12195): drop_caches: 1
bash (12650): drop_caches: 1
bash (13135): drop_caches: 1
bash (13600): drop_caches: 1
bash (14062): drop_caches: 1
bash (15508): drop_caches: 1
bash (17185): drop_caches: 1
bash (18770): drop_caches: 1
bash (20439): drop_caches: 1
bash (22136): drop_caches: 1
bash (23961): drop_caches: 1
bash (25863): drop_caches: 1
bash (27542): drop_caches: 1
bash (29172): drop_caches: 1
bash (30887): drop_caches: 1
bash (32533): drop_caches: 1
bash (1785): drop_caches: 1
bash (3477): drop_caches: 1
bash (5192): drop_caches: 1
bash (6768): drop_caches: 1
bash (8364): drop_caches: 1
bash (9961): drop_caches: 1
bash (11498): drop_caches: 1
bash (13159): drop_caches: 1
bash (14711): drop_caches: 1
bash (16332): drop_caches: 1
bash (17955): drop_caches: 1
bash (19654): drop_caches: 1
bash (21317): drop_caches: 1
bash (22920): drop_caches: 1
bash (24905): drop_caches: 1
bash (26718): drop_caches: 1
bash (28410): drop_caches: 1
bash (30030): drop_caches: 1
bash (31545): drop_caches: 1
bash (735): drop_caches: 1
bash (2307): drop_caches: 1
bash (3976): drop_caches: 1
bash (5493): drop_caches: 1
bash (7093): drop_caches: 1
bash (8681): drop_caches: 1
bash (10083): drop_caches: 1
bash (11514): drop_caches: 1
bash (12934): drop_caches: 1
bash (14519): drop_caches: 1
bash (16198): drop_caches: 1
bash (17897): drop_caches: 1
bash (19406): drop_caches: 1
bash (20902): drop_caches: 1
bash (22539): drop_caches: 1
bash (24153): drop_caches: 1
bash (26011): drop_caches: 1
bash (27632): drop_caches: 1
bash (29221): drop_caches: 1
bash (30735): drop_caches: 1
bash (32301): drop_caches: 1
bash (1392): drop_caches: 1
bash (2957): drop_caches: 1
bash (4606): drop_caches: 1
bash (6108): drop_caches: 1
bash (7593): drop_caches: 1
bash (9115): drop_caches: 1
bash (10558): drop_caches: 1
bash (12041): drop_caches: 1
bash (13565): drop_caches: 1
bash (15110): drop_caches: 1
bash (16423): drop_caches: 1
bash (17863): drop_caches: 1
bash (19412): drop_caches: 1
bash (21021): drop_caches: 1
bash (22622): drop_caches: 1
bash (24475): drop_caches: 1
bash (26220): drop_caches: 1
bash (27875): drop_caches: 1
bash (29513): drop_caches: 1
bash (30982): drop_caches: 1
bash (32523): drop_caches: 1
bash (1703): drop_caches: 1
bash (3249): drop_caches: 1
bash (4751): drop_caches: 1
bash (6270): drop_caches: 1
bash (7861): drop_caches: 1
bash (9293): drop_caches: 1
bash (10915): drop_caches: 1
bash (12361): drop_caches: 1
bash (13902): drop_caches: 1
bash (15491): drop_caches: 1
bash (16916): drop_caches: 1
bash (18431): drop_caches: 1
bash (20061): drop_caches: 1
bash (21668): drop_caches: 1
bash (23386): drop_caches: 1
bash (25233): drop_caches: 1
bash (26795): drop_caches: 1
bash (28618): drop_caches: 1
bash (30227): drop_caches: 1
bash (31676): drop_caches: 1
bash (720): drop_caches: 1
bash (2143): drop_caches: 1
bash (3651): drop_caches: 1
bash (5242): drop_caches: 1
bash (6736): drop_caches: 1
bash (8268): drop_caches: 1
bash (9682): drop_caches: 1
bash (11110): drop_caches: 1
bash (12581): drop_caches: 1
bash (14044): drop_caches: 1
bash (15533): drop_caches: 1
bash (16968): drop_caches: 1
bash (18533): drop_caches: 1
bash (20002): drop_caches: 1
bash (21493): drop_caches: 1
bash (23196): drop_caches: 1
bash (25010): drop_caches: 1
bash (26712): drop_caches: 1
bash (28332): drop_caches: 1
bash (29710): drop_caches: 1
bash (31290): drop_caches: 1
bash (321): drop_caches: 1
bash (1912): drop_caches: 1
bash (3516): drop_caches: 1
bash (5061): drop_caches: 1
bash (6636): drop_caches: 1
bash (8125): drop_caches: 1
bash (9617): drop_caches: 1
bash (11081): drop_caches: 1
bash (12688): drop_caches: 1
bash (14200): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 24.7M 551M 0 0 0 958
bash (15645): drop_caches: 1
lustre-mdt1 24.7M 551M 0 0 0 947
lustre-mdt1 24.7M 551M 0 0 0 976
lustre-mdt1 26.7M 549M 0 75 0 2.17M
lustre-mdt1 28.9M 547M 0 73 0 2.15M
lustre-mdt1 28.9M 547M 0 13 0 478K
lustre-mdt1 31.0M 545M 0 85 0 1.72M
lustre-mdt1 33.7M 542M 0 224 0 3.54M
lustre-mdt1 33.7M 542M 0 106 0 2.17M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a022067 PUD 31b1c1067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata floppy serio_raw i2c_core virtio_blk [last unloaded: libcfs]
CPU: 10 PID: 975 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e89c9d0 ti: ffff88031e8f4000 task.ti: ffff88031e8f4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031e8f7e50 EFLAGS: 00010246
RAX: ffff88031e8f7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331c93a98 RDI: 0000000000000063
RBP: ffff88031e8f7e50 R08: 0000000000000000 R09: 000000000000c52d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f3b8d69a740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031e910000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 58.3M 518M 0 149 0 1.32M
lustre-mdt1 58.4M 518M 0 117 0 2.23M
lustre-mdt1 58.5M 517M 0 262 0 3.92M
lustre-mdt1 60.6M 515M 0 179 0 3.12M
lustre-mdt1 58.7M 517M 0 173 0 1.59M
lustre-mdt1 57.8M 518M 0 353 0 6.12M
lustre-mdt1 59.8M 516M 0 155 0 3.13M
lustre-mdt1 59.8M 516M 0 362 0 4.64M
lustre-mdt1 59.2M 517M 0 251 0 3.27M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a7de067 PUD 31e637067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy i2c_core libata [last unloaded: libcfs]
CPU: 4 PID: 1005 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880319439280 ti: ffff8800b97d8000 task.ti: ffff8800b97d8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b97dbe50 EFLAGS: 00010246
RAX: ffff8800b97dbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b13a98 RDI: 0000000000000063
RBP: ffff8800b97dbe50 R08: 0000000000000000 R09: 0000000000002538
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fcdd2026740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031e8f2000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 20.2M 556M 0 0 0 984
lustre-mdt1 20.2M 556M 0 0 0 973
lustre-mdt1 20.2M 556M 0 0 0 973
lustre-mdt1 20.2M 556M 0 0 0 942
lustre-mdt1 20.2M 556M 0 0 0 857
lustre-mdt1 20.2M 556M 0 0 0 956
lustre-mdt1 20.2M 556M 0 0 0 952
lustre-mdt1 20.2M 556M 0 0 0 979
lustre-mdt1 20.2M 556M 0 0 0 935
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 327a69067 PUD 31999d067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk libata i2c_core serio_raw [last unloaded: libcfs]
CPU: 1 PID: 977 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031fc8a4f0 ti: ffff88031e764000 task.ti: ffff88031e764000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031e767e50 EFLAGS: 00010246
RAX: ffff88031e767fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031e767e50 R08: 0000000000000000 R09: 000000000000c379
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f5ca6396740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 0000000319246000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 49.3M 527M 0 230 0 3.49M
lustre-mdt1 49.1M 527M 0 238 0 3.72M
lustre-mdt1 49.2M 527M 0 260 0 4.34M
lustre-mdt1 48.7M 527M 0 264 0 3.66M
lustre-mdt1 47.7M 528M 0 250 0 3.61M
lustre-mdt1 47.2M 529M 0 199 0 3.95M
lustre-mdt1 48.5M 528M 0 130 0 2.98M
lustre-mdt1 48.5M 528M 0 60 0 1.92M
lustre-mdt1 48.5M 528M 0 346 0 3.70M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 327966067 PUD 31d93d067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm drm ata_generic pata_acpi drm_panel_orientation_quirks floppy ata_piix virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 7 PID: 984 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e57a4f0 ti: ffff880318800000 task.ti: ffff880318800000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880318803e50 EFLAGS: 00010246
RAX: ffff880318803fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff880318803e50 R08: 0000000000000000 R09: 0000000000008f48
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f6203fcc740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031def8000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 20.1M 556M 0 0 0 906
lustre-mdt1 20.1M 556M 0 1 0 1.78K
lustre-mdt1 20.1M 556M 0 0 0 986
lustre-mdt1 20.1M 556M 0 0 0 980
lustre-mdt1 20.1M 556M 0 0 0 970
lustre-mdt1 20.1M 556M 0 0 0 939
lustre-mdt1 20.1M 556M 0 0 0 845
lustre-mdt1 20.1M 556M 0 0 0 891
lustre-mdt1 20.1M 556M 0 0 0 984
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b999067 PUD 31991a067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 6 PID: 988 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031d953760 ti: ffff88031a250000 task.ti: ffff88031a250000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a253e50 EFLAGS: 00010246
RAX: ffff88031a253fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b93a98 RDI: 0000000000000063
RBP: ffff88031a253e50 R08: 0000000000000000 R09: 000000000000c292
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fd596cf3740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031c598000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 52.4M 524M 0 293 0 4.18M
lustre-mdt1 52.1M 524M 0 157 0 2.01M
lustre-mdt1 53.2M 523M 0 228 0 3.40M
lustre-mdt1 52.8M 523M 0 109 0 1.92M
lustre-mdt1 53.7M 522M 0 150 0 2.47M
lustre-mdt1 52.6M 523M 0 252 0 3.82M
lustre-mdt1 51.7M 524M 0 76 0 378K
lustre-mdt1 50.9M 525M 0 235 0 3.60M
lustre-mdt1 51.1M 525M 0 328 0 4.06M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a1cd067 PUD 31947d067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix i2c_core floppy virtio_blk serio_raw libata [last unloaded: libcfs]
CPU: 8 PID: 997 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031af0dc40 ti: ffff8800ba430000 task.ti: ffff8800ba430000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba433e50 EFLAGS: 00010246
RAX: ffff8800ba433fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba433e50 R08: 0000000000000000 R09: 0000000000003ef1
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fc459e7a740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ac2c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 53.5M 522M 0 165 0 2.99M
lustre-mdt1 52.8M 523M 0 259 0 3.68M
lustre-mdt1 53.8M 522M 0 285 0 4.02M
lustre-mdt1 55.5M 520M 0 342 0 4.25M
lustre-mdt1 54.5M 522M 0 181 0 2.22M
lustre-mdt1 54.2M 522M 0 412 0 5.46M
lustre-mdt1 54.1M 522M 0 364 0 5.01M
lustre-mdt1 54.9M 521M 0 175 0 2.54M
lustre-mdt1 53.0M 523M 0 292 0 3.03M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb06f067 PUD bb956067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm drm_panel_orientation_quirks pata_acpi ata_piix virtio_blk i2c_core serio_raw libata floppy [last unloaded: libcfs]
CPU: 11 PID: 981 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ec1a4f0 ti: ffff8803220d0000 task.ti: ffff8803220d0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803220d3e50 EFLAGS: 00010246
RAX: ffff8803220d3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8803220d3e50 R08: 0000000000000000 R09: 000000000000667f
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f06311ca740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000babe8000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x2c0000405 to 0x2c0000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 49.1M 527M 0 218 0 2.46M
lustre-mdt1 49.4M 527M 0 389 0 4.68M
lustre-mdt1 49.1M 527M 0 262 0 3.46M
lustre-mdt1 48.4M 528M 0 307 0 2.15M
lustre-mdt1 48.7M 527M 0 386 0 4.62M
lustre-mdt1 49.1M 527M 0 317 0 3.85M
lustre-mdt1 49.7M 526M 0 249 0 2.99M
lustre-mdt1 50.0M 526M 0 347 0 3.75M
lustre-mdt1 48.9M 527M 0 299 0 4.33M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b1d6067 PUD 31a97d067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix floppy serio_raw libata virtio_blk i2c_core [last unloaded: libcfs]
CPU: 13 PID: 1004 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8803191d3760 ti: ffff8800ba068000 task.ti: ffff8800ba068000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba06be50 EFLAGS: 00010246
RAX: ffff8800ba06bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba06be50 R08: 0000000000000000 R09: 00000000000020b8
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f57bd13e740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031f2a4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x280000405 to 0x280000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 43.5M 533M 0 0 0 977
lustre-mdt1 43.5M 533M 0 0 0 1000
lustre-mdt1 44.2M 532M 0 109 0 782K
lustre-mdt1 44.2M 532M 0 0 0 981
lustre-mdt1 44.2M 532M 0 30 0 729K
lustre-mdt1 45.0M 531M 0 58 0 145K
lustre-mdt1 45.0M 531M 0 0 0 990
lustre-mdt1 45.0M 531M 0 0 0 997
lustre-mdt1 45.0M 531M 0 0 0 991
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e8d6067 PUD 31f350067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw libata virtio_blk i2c_core floppy [last unloaded: libcfs]
CPU: 6 PID: 979 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031f755c40 ti: ffff88031ef34000 task.ti: ffff88031ef34000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031ef37e50 EFLAGS: 00010246
RAX: ffff88031ef37fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b93a98 RDI: 0000000000000063
RBP: ffff88031ef37e50 R08: 0000000000000000 R09: 0000000000000a92
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f391bc41740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031a07a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

bash (24786): drop_caches: 1
bash (25209): drop_caches: 1
bash (25706): drop_caches: 1
bash (26316): drop_caches: 1
bash (26826): drop_caches: 1
bash (27326): drop_caches: 1
bash (27735): drop_caches: 1
bash (28190): drop_caches: 1
bash (28693): drop_caches: 1
bash (29161): drop_caches: 1
bash (29644): drop_caches: 1
bash (30093): drop_caches: 1
bash (30656): drop_caches: 1
bash (31189): drop_caches: 1
bash (31739): drop_caches: 1
bash (32321): drop_caches: 1
bash (32749): drop_caches: 1
bash (730): drop_caches: 1
bash (1300): drop_caches: 1
bash (1905): drop_caches: 1
bash (2386): drop_caches: 1
bash (2878): drop_caches: 1
bash (3447): drop_caches: 1
bash (3979): drop_caches: 1
bash (4384): drop_caches: 1
bash (4890): drop_caches: 1
bash (5283): drop_caches: 1
bash (5806): drop_caches: 1
bash (6203): drop_caches: 1
bash (6575): drop_caches: 1
bash (6987): drop_caches: 1
bash (7417): drop_caches: 1
bash (8150): drop_caches: 1
bash (9596): drop_caches: 1
bash (11175): drop_caches: 1
bash (13051): drop_caches: 1
bash (14782): drop_caches: 1
bash (16545): drop_caches: 1
bash (18191): drop_caches: 1
bash (19839): drop_caches: 1
bash (21522): drop_caches: 1
bash (23098): drop_caches: 1
bash (24631): drop_caches: 1
bash (26222): drop_caches: 1
bash (27779): drop_caches: 1
bash (29282): drop_caches: 1
bash (30838): drop_caches: 1
bash (32401): drop_caches: 1
bash (1514): drop_caches: 1
bash (2978): drop_caches: 1
bash (4553): drop_caches: 1
bash (6041): drop_caches: 1
bash (7709): drop_caches: 1
bash (9311): drop_caches: 1
bash (10910): drop_caches: 1
bash (12757): drop_caches: 1
bash (14630): drop_caches: 1
bash (16356): drop_caches: 1
bash (17994): drop_caches: 1
bash (19637): drop_caches: 1
bash (21191): drop_caches: 1
bash (22691): drop_caches: 1
bash (24063): drop_caches: 1
bash (25589): drop_caches: 1
bash (27045): drop_caches: 1
bash (28688): drop_caches: 1
bash (30316): drop_caches: 1
bash (31874): drop_caches: 1
bash (1020): drop_caches: 1
bash (2414): drop_caches: 1
bash (4008): drop_caches: 1
bash (5535): drop_caches: 1
bash (7175): drop_caches: 1
bash (8664): drop_caches: 1
bash (10256): drop_caches: 1
bash (11682): drop_caches: 1
bash (13554): drop_caches: 1
bash (15305): drop_caches: 1
bash (17062): drop_caches: 1
bash (18665): drop_caches: 1
bash (20180): drop_caches: 1
bash (21672): drop_caches: 1
bash (23103): drop_caches: 1
bash (24639): drop_caches: 1
bash (26276): drop_caches: 1
bash (27783): drop_caches: 1
bash (29262): drop_caches: 1
bash (30726): drop_caches: 1
bash (32112): drop_caches: 1
bash (1162): drop_caches: 1
bash (2504): drop_caches: 1
bash (4024): drop_caches: 1
bash (5525): drop_caches: 1
bash (7012): drop_caches: 1
bash (8639): drop_caches: 1
bash (10244): drop_caches: 1
bash (11760): drop_caches: 1
bash (13512): drop_caches: 1
bash (15360): drop_caches: 1
bash (16899): drop_caches: 1
bash (18471): drop_caches: 1
bash (19845): drop_caches: 1
bash (21351): drop_caches: 1
bash (22792): drop_caches: 1
bash (24291): drop_caches: 1
bash (25691): drop_caches: 1
bash (27239): drop_caches: 1
bash (28771): drop_caches: 1
bash (30263): drop_caches: 1
bash (31735): drop_caches: 1
bash (717): drop_caches: 1
bash (2391): drop_caches: 1
bash (3939): drop_caches: 1
bash (5578): drop_caches: 1
bash (7230): drop_caches: 1
bash (8799): drop_caches: 1
bash (10300): drop_caches: 1
bash (11845): drop_caches: 1
bash (13911): drop_caches: 1
bash (15614): drop_caches: 1
bash (17318): drop_caches: 1
bash (18885): drop_caches: 1
bash (20467): drop_caches: 1
bash (21935): drop_caches: 1
bash (23387): drop_caches: 1
bash (24826): drop_caches: 1
bash (26345): drop_caches: 1
bash (27850): drop_caches: 1
bash (29427): drop_caches: 1
bash (30889): drop_caches: 1
bash (32340): drop_caches: 1
bash (1509): drop_caches: 1
bash (3033): drop_caches: 1
bash (4484): drop_caches: 1
bash (6066): drop_caches: 1
bash (7640): drop_caches: 1
bash (9135): drop_caches: 1
bash (10584): drop_caches: 1
bash (12281): drop_caches: 1
bash (14077): drop_caches: 1
bash (16015): drop_caches: 1
bash (17777): drop_caches: 1
bash (19426): drop_caches: 1
bash (20862): drop_caches: 1
bash (22444): drop_caches: 1
bash (24110): drop_caches: 1
bash (25612): drop_caches: 1
bash (27054): drop_caches: 1
bash (28536): drop_caches: 1
bash (30124): drop_caches: 1
bash (31614): drop_caches: 1
bash (595): drop_caches: 1
bash (2316): drop_caches: 1
bash (3853): drop_caches: 1
bash (5408): drop_caches: 1
bash (6956): drop_caches: 1
bash (8487): drop_caches: 1
bash (10084): drop_caches: 1
bash (11557): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 43.5M 533M 0 0 0 973
lustre-mdt1 43.5M 533M 0 0 0 987
lustre-mdt1 43.5M 533M 0 0 0 983
lustre-mdt1 43.5M 533M 0 0 0 984
lustre-mdt1 43.5M 533M 0 0 0 990
lustre-mdt1 43.5M 533M 0 0 0 989
lustre-mdt1 43.5M 533M 0 0 0 990
lustre-mdt1 43.5M 533M 0 0 0 973
lustre-mdt1 43.5M 533M 0 0 0 968
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 318dd3067 PUD 31e4eb067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 6 PID: 1007 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e5bdc40 ti: ffff88031fd68000 task.ti: ffff88031fd68000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fd6be50 EFLAGS: 00010246
RAX: ffff88031fd6bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b93a98 RDI: 0000000000000063
RBP: ffff88031fd6be50 R08: 0000000000000000 R09: 00000000000037a5
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f5c27dfd740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031e550000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000405 to 0x240000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 50.6M 525M 0 320 0 4.18M
lustre-mdt1 49.0M 527M 0 402 0 3.88M
lustre-mdt1 49.2M 527M 0 406 0 4.61M
lustre-mdt1 50.4M 526M 0 294 0 2.61M
lustre-mdt1 50.4M 526M 0 411 0 4.54M
lustre-mdt1 50.6M 525M 0 357 0 4.19M
lustre-mdt1 49.0M 527M 0 443 0 4.89M
lustre-mdt1 49.5M 526M 0 550 0 6.57M
lustre-mdt1 50.5M 525M 0 245 0 3.10M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD ba6c4067 PUD ba6c6067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw floppy i2c_core libata [last unloaded: libcfs]
CPU: 8 PID: 992 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031a99dc40 ti: ffff88031aa34000 task.ti: ffff88031aa34000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031aa37e50 EFLAGS: 00010246
RAX: ffff88031aa37fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031aa37e50 R08: 0000000000000000 R09: 000000000000210c
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f380865a740(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000b93bc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 48.6M 527M 0 169 0 1.40M
lustre-mdt1 49.2M 527M 0 209 0 3.29M
lustre-mdt1 49.2M 527M 0 3 0 15.5K
lustre-mdt1 49.7M 526M 0 295 0 4.21M
lustre-mdt1 49.1M 527M 0 323 0 3.28M
lustre-mdt1 49.2M 527M 0 182 0 2.65M
lustre-mdt1 49.9M 526M 0 237 0 3.08M
lustre-mdt1 48.9M 527M 0 223 0 3.57M
lustre-mdt1 50.3M 526M 0 41 0 95.1K
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31ced7067 PUD 31abdc067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm libata drm_panel_orientation_quirks floppy serio_raw virtio_blk i2c_core [last unloaded: libcfs]
CPU: 5 PID: 1001 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800ba510010 ti: ffff8800ba7e4000 task.ti: ffff8800ba7e4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba7e7e50 EFLAGS: 00010246
RAX: ffff8800ba7e7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba7e7e50 R08: 0000000000000000 R09: 00000000000020cf
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f8334a1e740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000003190ba000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 52.2M 524M 0 524 0 5.23M
lustre-mdt1 51.5M 524M 0 241 0 3.34M
lustre-mdt1 51.2M 525M 0 194 0 1.29M
lustre-mdt1 52.8M 523M 0 285 0 2.94M
lustre-mdt1 52.6M 523M 0 606 0 5.80M
lustre-mdt1 51.6M 524M 0 322 0 3.77M
lustre-mdt1 52.5M 524M 0 291 0 3.36M
lustre-mdt1 53.4M 523M 0 363 0 3.90M
lustre-mdt1 54.4M 522M 0 257 0 3.27M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b941067 PUD 31a4e4067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy libata serio_raw virtio_blk i2c_core [last unloaded: libcfs]
CPU: 7 PID: 980 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b3f5c40 ti: ffff8803214c8000 task.ti: ffff8803214c8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803214cbe50 EFLAGS: 00010246
RAX: ffff8803214cbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331bd3a98 RDI: 0000000000000063
RBP: ffff8803214cbe50 R08: 0000000000000000 R09: 000000000000637d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007ff371474740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b8bc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 52.1M 524M 0 228 0 4.31M
lustre-mdt1 52.0M 524M 0 218 0 4.59M
lustre-mdt1 54.1M 522M 0 160 0 3.46M
lustre-mdt1 54.9M 521M 0 222 0 4.64M
lustre-mdt1 52.7M 523M 0 463 0 6.81M
lustre-mdt1 53.5M 522M 0 129 0 3.31M
lustre-mdt1 56.5M 519M 0 283 0 6.71M
lustre-mdt1 56.8M 519M 0 242 0 4.72M
lustre-mdt1 58.6M 517M 0 293 0 6.15M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31eccf067 PUD 327984067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs]
CPU: 11 PID: 986 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e91a4f0 ti: ffff88031b930000 task.ti: ffff88031b930000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b933e50 EFLAGS: 00010246
RAX: ffff88031b933fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331cd3a98 RDI: 0000000000000063
RBP: ffff88031b933e50 R08: 0000000000000000 R09: 00000000000023db
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fa337a38740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b2f2000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000: already connected client lustre-MDT0002-mdtlov_UUID (at 0@lo) with handle 0x55a04b8fe91ed812. Rejecting client with the same UUID trying to reconnect with handle 0xf01a36916a8136e2
Lustre: lustre-OST0000: already connected client lustre-MDT0002-mdtlov_UUID (at 0@lo) with handle 0x55a04b8fe91ed812. Rejecting client with the same UUID trying to reconnect with handle 0xf01a36916a8136e2
Lustre: Skipped 2 previous similar messages
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 59.6M 516M 0 27 0 402K
lustre-mdt1 59.0M 517M 0 57 0 418K
lustre-mdt1 59.0M 517M 0 35 0 594K
lustre-mdt1 58.0M 518M 0 28 0 67.0K
lustre-mdt1 58.0M 518M 0 1 0 1.25K
lustre-mdt1 58.0M 518M 0 60 0 750K
lustre-mdt1 58.0M 518M 0 5 0 6.13K
lustre-mdt1 58.0M 518M 0 10 0 88.2K
lustre-mdt1 57.3M 519M 0 52 0 561K
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31d266067 PUD 31e7c4067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks i2c_core virtio_blk serio_raw libata floppy [last unloaded: libcfs]
CPU: 9 PID: 987 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e4e1280 ti: ffff88031a2a4000 task.ti: ffff88031a2a4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a2a7e50 EFLAGS: 00010246
RAX: ffff88031a2a7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031a2a7e50 R08: 0000000000000000 R09: 00000000000008a3
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f00fe9e8740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031e852000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 56.4M 520M 0 401 0 4.61M
lustre-mdt1 57.2M 519M 0 442 0 6.65M
lustre-mdt1 55.7M 520M 0 465 0 7.67M
lustre-mdt1 56.8M 519M 0 294 0 4.52M
lustre-mdt1 53.8M 522M 0 453 0 4.68M
lustre-mdt1 55.0M 521M 0 371 0 6.26M
lustre-mdt1 56.6M 519M 0 384 0 6.19M
lustre-mdt1 56.6M 519M 0 585 0 9.08M
lustre-mdt1 57.3M 519M 0 533 0 9.27M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3191d3067 PUD 31f2e0067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk i2c_core serio_raw libata floppy [last unloaded: libcfs]
CPU: 13 PID: 976 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031fd49280 ti: ffff880319a7c000 task.ti: ffff880319a7c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880319a7fe50 EFLAGS: 00010246
RAX: ffff880319a7ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d53a98 RDI: 0000000000000063
RBP: ffff880319a7fe50 R08: 0000000000000000 R09: 00000000000008af
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f4935073740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 00000003220cc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x300000405 to 0x300000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 52.6M 523M 0 140 0 1.92M
lustre-mdt1 52.8M 523M 0 403 0 4.10M
lustre-mdt1 51.8M 524M 0 470 0 4.17M
lustre-mdt1 50.5M 525M 0 471 0 5.02M
lustre-mdt1 51.8M 524M 0 219 0 3.29M
lustre-mdt1 50.1M 526M 0 175 0 2.15M
lustre-mdt1 50.6M 525M 0 335 0 3.56M
lustre-mdt1 51.5M 525M 0 419 0 4.93M
lustre-mdt1 51.5M 525M 0 384 0 4.71M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31d249067 PUD 31d513067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks floppy serio_raw ata_piix virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 11 PID: 976 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ede1280 ti: ffff88031ef20000 task.ti: ffff88031ef20000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031ef23e50 EFLAGS: 00010246
RAX: ffff88031ef23fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331cd3a98 RDI: 0000000000000063
RBP: ffff88031ef23e50 R08: 0000000000000000 R09: 000000000000bcf9
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007feeb5212740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ae98000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 55.9M 520M 0 333 0 5.13M
lustre-mdt1 52.9M 523M 0 390 0 5.32M
lustre-mdt1 53.2M 523M 0 327 0 5.36M
lustre-mdt1 54.1M 522M 0 151 0 3.34M
lustre-mdt1 55.9M 520M 0 407 0 6.17M
lustre-mdt1 55.6M 520M 0 527 0 7.48M
lustre-mdt1 52.1M 524M 0 393 0 5.40M
lustre-mdt1 53.1M 523M 0 221 0 4.40M
lustre-mdt1 54.3M 522M 0 134 0 2.23M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31c512067 PUD 31c513067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs]
CPU: 0 PID: 986 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031f758010 ti: ffff88031d4d8000 task.ti: ffff88031d4d8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031d4dbe50 EFLAGS: 00010246
RAX: ffff88031d4dbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a13a98 RDI: 0000000000000063
RBP: ffff88031d4dbe50 R08: 0000000000000000 R09: 0000000000006437
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f007285d740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319dc8000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff88003e181bf8: disconnect after 22s idle
Lustre: Skipped 3 previous similar messages
lustre-mdt1 37.9M 538M 3 79 8.75K 277K
lustre-mdt1 40.4M 536M 0 202 0 2.53M
lustre-mdt1 42.7M 533M 0 192 0 2.45M
lustre-mdt1 45.2M 531M 0 361 0 4.30M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31cdff067 PUD 31a530067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw libata i2c_core [last unloaded: libcfs]
CPU: 14 PID: 1003 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b8e51280 ti: ffff880320334000 task.ti: ffff880320334000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880320337e50 EFLAGS: 00010246
RAX: ffff880320337fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff880320337e50 R08: 0000000000000000 R09: 0000000000002036
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f2b5c32a740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031afd8000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

bash (18846): drop_caches: 1
bash (19362): drop_caches: 1
bash (19923): drop_caches: 1
bash (20403): drop_caches: 1
bash (20806): drop_caches: 1
bash (21200): drop_caches: 1
bash (21685): drop_caches: 1
bash (22376): drop_caches: 1
bash (22992): drop_caches: 1
bash (23563): drop_caches: 1
bash (24100): drop_caches: 1
bash (24641): drop_caches: 1
bash (25140): drop_caches: 1
bash (26080): drop_caches: 1
bash (27873): drop_caches: 1
bash (29617): drop_caches: 1
bash (31552): drop_caches: 1
bash (916): drop_caches: 1
bash (2712): drop_caches: 1
bash (4535): drop_caches: 1
bash (6424): drop_caches: 1
bash (8327): drop_caches: 1
bash (10218): drop_caches: 1
bash (12202): drop_caches: 1
bash (14102): drop_caches: 1
bash (16180): drop_caches: 1
bash (18273): drop_caches: 1
bash (20332): drop_caches: 1
bash (22328): drop_caches: 1
bash (24424): drop_caches: 1
bash (26292): drop_caches: 1
bash (28247): drop_caches: 1
bash (30013): drop_caches: 1
bash (31876): drop_caches: 1
bash (1301): drop_caches: 1
bash (3203): drop_caches: 1
bash (5184): drop_caches: 1
bash (7031): drop_caches: 1
bash (8874): drop_caches: 1
bash (10776): drop_caches: 1
bash (12622): drop_caches: 1
bash (14543): drop_caches: 1
bash (16611): drop_caches: 1
bash (18748): drop_caches: 1
bash (20666): drop_caches: 1
bash (22415): drop_caches: 1
bash (24411): drop_caches: 1
bash (26280): drop_caches: 1
bash (28106): drop_caches: 1
bash (29954): drop_caches: 1
bash (31901): drop_caches: 1
bash (1402): drop_caches: 1
bash (3329): drop_caches: 1
bash (5060): drop_caches: 1
bash (6965): drop_caches: 1
bash (8912): drop_caches: 1
bash (10720): drop_caches: 1
bash (12551): drop_caches: 1
bash (14457): drop_caches: 1
bash (16707): drop_caches: 1
bash (18819): drop_caches: 1
bash (20858): drop_caches: 1
bash (22812): drop_caches: 1
bash (24823): drop_caches: 1
bash (26722): drop_caches: 1
bash (28537): drop_caches: 1
bash (30473): drop_caches: 1
bash (32261): drop_caches: 1
bash (1680): drop_caches: 1
bash (3860): drop_caches: 1
bash (5836): drop_caches: 1
bash (7848): drop_caches: 1
bash (9821): drop_caches: 1
bash (11933): drop_caches: 1
bash (13860): drop_caches: 1
bash (15652): drop_caches: 1
bash (17963): drop_caches: 1
bash (19947): drop_caches: 1
bash (21804): drop_caches: 1
bash (23609): drop_caches: 1
bash (25469): drop_caches: 1
bash (27326): drop_caches: 1
bash (29128): drop_caches: 1
bash (30851): drop_caches: 1
bash (32716): drop_caches: 1
bash (2279): drop_caches: 1
bash (4213): drop_caches: 1
bash (6091): drop_caches: 1
bash (7883): drop_caches: 1
bash (9839): drop_caches: 1
bash (11704): drop_caches: 1
bash (13539): drop_caches: 1
bash (15542): drop_caches: 1
bash (18006): drop_caches: 1
bash (20133): drop_caches: 1
bash (22095): drop_caches: 1
bash (24103): drop_caches: 1
bash (26087): drop_caches: 1
bash (28108): drop_caches: 1
bash (30067): drop_caches: 1
bash (32090): drop_caches: 1
bash (1600): drop_caches: 1
bash (3673): drop_caches: 1
bash (5894): drop_caches: 1
bash (7862): drop_caches: 1
bash (9736): drop_caches: 1
bash (11807): drop_caches: 1
bash (13744): drop_caches: 1
bash (16220): drop_caches: 1
bash (18407): drop_caches: 1
bash (20525): drop_caches: 1
bash (22424): drop_caches: 1
bash (24540): drop_caches: 1
bash (26378): drop_caches: 1
bash (28233): drop_caches: 1
bash (30187): drop_caches: 1
bash (32215): drop_caches: 1
bash (1809): drop_caches: 1
bash (3750): drop_caches: 1
bash (5603): drop_caches: 1
bash (7517): drop_caches: 1
bash (9412): drop_caches: 1
bash (11212): drop_caches: 1
bash (12984): drop_caches: 1
bash (15034): drop_caches: 1
bash (17291): drop_caches: 1
bash (19369): drop_caches: 1
bash (21421): drop_caches: 1
bash (23384): drop_caches: 1
bash (25412): drop_caches: 1
bash (27295): drop_caches: 1
bash (29300): drop_caches: 1
bash (31143): drop_caches: 1
bash (602): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 29.3M 547M 0 133 0 2.61M
lustre-mdt1 32.7M 543M 0 286 0 5.60M
lustre-mdt1 31.7M 544M 0 306 0 4.97M
lustre-mdt1 32.6M 543M 0 144 0 2.71M
lustre-mdt1 36.2M 540M 0 233 0 5.14M
lustre-mdt1 35.1M 541M 0 79 0 799K
lustre-mdt1 36.5M 540M 0 198 0 2.16M
lustre-mdt1 28.4M 548M 0 293 0 2.20M
lustre-mdt1 28.4M 548M 0 0 0 993
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31bf9c067 PUD 31b385067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks i2c_core virtio_blk serio_raw floppy libata [last unloaded: libcfs]
CPU: 1 PID: 998 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880319d624f0 ti: ffff88031ec8c000 task.ti: ffff88031ec8c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031ec8fe50 EFLAGS: 00010246
RAX: ffff88031ec8ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031ec8fe50 R08: 0000000000000000 R09: 000000000000a603
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f1afd124740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031d152000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 50.2M 526M 0 297 0 4.08M
lustre-mdt1 51.0M 525M 0 160 0 3.12M
lustre-mdt1 51.6M 524M 0 137 0 2.72M
lustre-mdt1 52.7M 523M 0 214 0 4.20M
lustre-mdt1 52.5M 523M 0 273 0 3.92M
lustre-mdt1 51.3M 525M 0 118 0 1.28M
lustre-mdt1 50.9M 525M 0 270 0 4.18M
lustre-mdt1 51.4M 525M 0 446 0 6.63M
lustre-mdt1 52.3M 524M 0 146 0 3.27M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b0de067 PUD 31f25a067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ata_piix ttm drm drm_panel_orientation_quirks libata serio_raw virtio_blk i2c_core floppy [last unloaded: libcfs]
CPU: 14 PID: 992 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031dcd24f0 ti: ffff88031f12c000 task.ti: ffff88031f12c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031f12fe50 EFLAGS: 00010246
RAX: ffff88031f12ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff88031f12fe50 R08: 0000000000000000 R09: 000000000000089b
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007faf4f587740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000320b12000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 48.8M 527M 0 337 0 3.39M
lustre-mdt1 50.4M 526M 0 306 0 3.29M
lustre-mdt1 50.7M 525M 0 149 0 1.60M
lustre-mdt1 50.9M 525M 0 571 0 5.77M
lustre-mdt1 49.5M 526M 0 365 0 4.01M
lustre-mdt1 50.3M 526M 0 373 0 4.59M
lustre-mdt1 51.5M 524M 0 261 0 3.17M
lustre-mdt1 52.2M 524M 0 279 0 3.23M
lustre-mdt1 51.4M 525M 0 348 0 3.95M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31f2cc067 PUD 3194e3067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs]
CPU: 13 PID: 984 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800ba389280 ti: ffff88031dbc4000 task.ti: ffff88031dbc4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031dbc7e50 EFLAGS: 00010246
RAX: ffff88031dbc7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d53a98 RDI: 0000000000000063
RBP: ffff88031dbc7e50 R08: 0000000000000000 R09: 0000000000004e0f
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f56bccfb740(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031f38e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.6M 521M 0 381 0 3.91M
lustre-mdt1 55.7M 520M 0 553 0 6.94M
lustre-mdt1 56.5M 519M 0 256 0 3.48M
lustre-mdt1 57.5M 518M 0 345 0 5.79M
lustre-mdt1 57.0M 519M 0 415 0 3.74M
lustre-mdt1 54.2M 522M 0 384 0 4.70M
lustre-mdt1 53.3M 523M 0 378 0 5.14M
lustre-mdt1 54.4M 522M 0 334 0 4.37M
lustre-mdt1 52.3M 524M 0 477 0 4.20M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31ae8b067 PUD 31ae8a067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix floppy virtio_blk drm_panel_orientation_quirks libata i2c_core serio_raw [last unloaded: libcfs]
CPU: 4 PID: 946 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880320331280 ti: ffff88031d55c000 task.ti: ffff88031d55c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031d55fe50 EFLAGS: 00010246
RAX: ffff88031d55ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b13a98 RDI: 0000000000000063
RBP: ffff88031d55fe50 R08: 0000000000000000 R09: 0000000000007947
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f1998843740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031f28a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 48.8M 527M 0 256 0 2.63M
lustre-mdt1 48.3M 528M 0 195 0 2.15M
lustre-mdt1 48.4M 528M 0 210 0 2.07M
lustre-mdt1 49.3M 527M 0 319 0 3.50M
lustre-mdt1 49.2M 527M 0 402 0 3.35M
lustre-mdt1 49.3M 527M 0 251 0 2.66M
lustre-mdt1 48.4M 528M 0 304 0 3.13M
lustre-mdt1 48.4M 528M 0 120 0 734K
lustre-mdt1 48.9M 527M 0 399 0 4.40M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a60e067 PUD 31cc95067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy virtio_blk i2c_core libata [last unloaded: libcfs]
CPU: 11 PID: 1001 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800ba93a4f0 ti: ffff88031fdc0000 task.ti: ffff88031fdc0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fdc3e50 EFLAGS: 00010246
RAX: ffff88031fdc3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331cd3a98 RDI: 0000000000000063
RBP: ffff88031fdc3e50 R08: 0000000000000000 R09: 00000000000036b6
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007eff986b9740(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031a504000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 50.2M 526M 0 397 0 4.63M
lustre-mdt1 51.6M 524M 0 363 0 4.63M
lustre-mdt1 52.0M 524M 0 338 0 4.40M
lustre-mdt1 53.3M 523M 0 322 0 5.83M
lustre-mdt1 54.9M 521M 0 450 0 4.37M
lustre-mdt1 55.1M 521M 0 321 0 4.95M
lustre-mdt1 55.2M 521M 0 260 0 3.71M
lustre-mdt1 55.2M 521M 0 323 0 4.46M
lustre-mdt1 51.6M 524M 0 462 0 5.37M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b801067 PUD 31b800067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk floppy serio_raw libata i2c_core [last unloaded: libcfs]
CPU: 5 PID: 997 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b0e8010 ti: ffff88031fe38000 task.ti: ffff88031fe38000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fe3be50 EFLAGS: 00010246
RAX: ffff88031fe3bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b53a98 RDI: 0000000000000063
RBP: ffff88031fe3be50 R08: 0000000000000000 R09: 0000000000007171
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f5c509c2740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ae4a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 22.8M 553M 0 0 0 965
lustre-mdt1 22.8M 553M 0 0 0 892
lustre-mdt1 22.8M 553M 0 0 0 964
lustre-mdt1 22.8M 553M 0 0 0 942
lustre-mdt1 22.8M 553M 0 0 0 958
lustre-mdt1 22.8M 553M 0 0 0 927
lustre-mdt1 22.8M 553M 0 1 0 1.84K
lustre-mdt1 22.8M 553M 0 0 0 980
lustre-mdt1 22.8M 553M 0 0 0 968
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31fdd2067 PUD 319497067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw i2c_core virtio_blk floppy [last unloaded: libcfs]
CPU: 3 PID: 990 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880319c6dc40 ti: ffff88031ab0c000 task.ti: ffff88031ab0c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031ab0fe50 EFLAGS: 00010246
RAX: ffff88031ab0ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031ab0fe50 R08: 0000000000000000 R09: 0000000000003da0
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fa38df9d740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000ba010000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 63.0M 513M 0 75 0 901K
lustre-mdt1 62.4M 514M 0 188 0 2.40M
lustre-mdt1 62.4M 514M 0 303 0 5.16M
lustre-mdt1 60.2M 516M 0 390 0 4.60M
lustre-mdt1 59.9M 516M 0 241 0 3.67M
lustre-mdt1 61.8M 514M 0 230 0 4.89M
lustre-mdt1 60.5M 516M 0 318 0 3.82M
lustre-mdt1 62.3M 514M 0 321 0 5.57M
lustre-mdt1 64.0M 512M 0 139 0 2.96M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3214c9067 PUD 319757067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk i2c_core floppy libata [last unloaded: libcfs]
CPU: 10 PID: 1002 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031f2c5c40 ti: ffff8800bafd0000 task.ti: ffff8800bafd0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bafd3e50 EFLAGS: 00010246
RAX: ffff8800bafd3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331c93a98 RDI: 0000000000000063
RBP: ffff8800bafd3e50 R08: 0000000000000000 R09: 0000000000003591
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f19534c1740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319c42000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 48.1M 528M 0 245 0 3.06M
lustre-mdt1 49.5M 526M 0 4 0 4.50K
lustre-mdt1 49.8M 526M 0 400 0 5.57M
lustre-mdt1 48.6M 527M 0 158 0 3.65M
lustre-mdt1 47.5M 528M 0 164 0 1.46M
lustre-mdt1 48.8M 527M 0 143 0 2.94M
lustre-mdt1 49.2M 527M 0 206 0 4.49M
lustre-mdt1 50.2M 526M 0 112 0 2.15M
lustre-mdt1 48.7M 527M 0 123 0 2.16M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e3ff067 PUD 31b006067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks virtio_blk serio_raw floppy libata i2c_core [last unloaded: libcfs]
CPU: 14 PID: 1011 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8803223aa4f0 ti: ffff88031a558000 task.ti: ffff88031a558000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a55be50 EFLAGS: 00010246
RAX: ffff88031a55bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff88031a55be50 R08: 0000000000000000 R09: 0000000000002406
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f78468a3740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ef76000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 22.6M 553M 0 0 0 979
lustre-mdt1 22.6M 553M 0 0 0 977
lustre-mdt1 22.6M 553M 0 0 0 979
lustre-mdt1 22.6M 553M 0 0 0 987
lustre-mdt1 22.6M 553M 0 0 0 986
lustre-mdt1 22.6M 553M 0 0 0 969
lustre-mdt1 22.6M 553M 0 0 0 966
lustre-mdt1 22.6M 553M 0 0 0 984
lustre-mdt1 22.6M 553M 0 0 0 967
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD ba9cf067 PUD bad56067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks serio_raw virtio_blk floppy ata_piix i2c_core libata [last unloaded: libcfs]
CPU: 12 PID: 1007 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880327a749d0 ti: ffff8800b93d4000 task.ti: ffff8800b93d4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b93d7e50 EFLAGS: 00010246
RAX: ffff8800b93d7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d13a98 RDI: 0000000000000063
RBP: ffff8800b93d7e50 R08: 0000000000000000 R09: 0000000000008f77
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fb35d804740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000baaa4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.9M 524M 0 336 0 4.24M
lustre-mdt1 52.1M 524M 0 278 0 3.51M
lustre-mdt1 52.5M 523M 0 270 0 4.10M
lustre-mdt1 54.1M 522M 0 190 0 2.95M
lustre-mdt1 54.5M 522M 0 325 0 4.78M
lustre-mdt1 51.6M 524M 0 457 0 6.17M
lustre-mdt1 50.8M 525M 0 136 0 2.43M
lustre-mdt1 52.3M 524M 0 587 0 7.26M
lustre-mdt1 53.8M 522M 0 484 0 5.42M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b01b067 PUD 320457067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix i2c_core serio_raw virtio_blk floppy libata [last unloaded: libcfs]
CPU: 14 PID: 1012 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031bef49d0 ti: ffff88031d1dc000 task.ti: ffff88031d1dc000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031d1dfe50 EFLAGS: 00010246
RAX: ffff88031d1dffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031d1dfe50 R08: 0000000000000000 R09: 000000000000a621
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f20a76f3740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b2f4000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 47.0M 529M 0 354 0 3.07M
lustre-mdt1 41.9M 534M 0 71 0 700K
lustre-mdt1 41.9M 534M 0 0 0 970
lustre-mdt1 41.9M 534M 0 0 0 973
lustre-mdt1 41.9M 534M 0 0 0 985
lustre-mdt1 42.6M 533M 0 99 0 800K
lustre-mdt1 42.6M 533M 0 0 0 994
lustre-mdt1 42.6M 533M 0 0 0 990
lustre-mdt1 42.6M 533M 0 0 0 998
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31ef80067 PUD 31f18f067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix floppy serio_raw drm_panel_orientation_quirks libata virtio_blk i2c_core [last unloaded: libcfs]
CPU: 6 PID: 987 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b111280 ti: ffff880319710000 task.ti: ffff880319710000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880319713e50 EFLAGS: 00010246
RAX: ffff880319713fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff880319713e50 R08: 0000000000000000 R09: 0000000000006380
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f37523ec740(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031975c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x280000405 to 0x280000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 53.2M 523M 0 236 0 3.29M
lustre-mdt1 51.2M 525M 0 586 0 6.33M
lustre-mdt1 52.3M 524M 0 219 0 3.15M
lustre-mdt1 53.0M 523M 0 361 0 4.40M
lustre-mdt1 52.8M 523M 0 512 0 6.09M
lustre-mdt1 54.1M 522M 0 551 0 5.02M
lustre-mdt1 53.9M 522M 0 523 0 6.24M
lustre-mdt1 54.6M 521M 0 242 0 3.21M
lustre-mdt1 55.3M 521M 0 483 0 5.82M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3198d3067 PUD 31bb2e067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core floppy libata [last unloaded: libcfs]
CPU: 2 PID: 960 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b8569280 ti: ffff8803220c0000 task.ti: ffff8803220c0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8803220c3e50 EFLAGS: 00010246
RAX: ffff8803220c3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8803220c3e50 R08: 0000000000000000 R09: 0000000000002063
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f663d6f1740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031a1c0000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802fa3cdd28: disconnect after 23s idle
Lustre: Skipped 2 previous similar messages
bash (28431): drop_caches: 1
bash (31045): drop_caches: 1
bash (1115): drop_caches: 1
bash (3550): drop_caches: 1
bash (5935): drop_caches: 1
bash (8150): drop_caches: 1
bash (10575): drop_caches: 1
bash (12783): drop_caches: 1
bash (14847): drop_caches: 1
bash (17022): drop_caches: 1
bash (19367): drop_caches: 1
bash (21542): drop_caches: 1
bash (23757): drop_caches: 1
bash (25887): drop_caches: 1
bash (28153): drop_caches: 1
bash (30402): drop_caches: 1
bash (351): drop_caches: 1
bash (2936): drop_caches: 1
bash (5105): drop_caches: 1
bash (7284): drop_caches: 1
bash (9954): drop_caches: 1
bash (12557): drop_caches: 1
bash (14994): drop_caches: 1
bash (17463): drop_caches: 1
bash (19699): drop_caches: 1
bash (22259): drop_caches: 1
bash (24464): drop_caches: 1
bash (27196): drop_caches: 1
bash (29770): drop_caches: 1
bash (32342): drop_caches: 1
bash (1908): drop_caches: 1
Lustre: 8502:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (4409): drop_caches: 1
bash (6803): drop_caches: 1
bash (9203): drop_caches: 1
bash (11699): drop_caches: 1
bash (13731): drop_caches: 1
bash (15797): drop_caches: 1
bash (17948): drop_caches: 1
bash (20086): drop_caches: 1
bash (22531): drop_caches: 1
bash (24748): drop_caches: 1
bash (27143): drop_caches: 1
bash (29711): drop_caches: 1
bash (32312): drop_caches: 1
bash (2265): drop_caches: 1
bash (4597): drop_caches: 1
bash (6950): drop_caches: 1
bash (9841): drop_caches: 1
bash (12264): drop_caches: 1
bash (14729): drop_caches: 1
bash (17103): drop_caches: 1
bash (19294): drop_caches: 1
bash (21806): drop_caches: 1
bash (24056): drop_caches: 1
bash (26646): drop_caches: 1
bash (29138): drop_caches: 1
bash (31459): drop_caches: 1
bash (1405): drop_caches: 1
bash (4050): drop_caches: 1
bash (6500): drop_caches: 1
bash (8770): drop_caches: 1
bash (11165): drop_caches: 1
bash (13382): drop_caches: 1
bash (15504): drop_caches: 1
bash (18115): drop_caches: 1
bash (20551): drop_caches: 1
bash (22792): drop_caches: 1
bash (25032): drop_caches: 1
bash (27226): drop_caches: 1
bash (29800): drop_caches: 1
bash (32457): drop_caches: 1
bash (2296): drop_caches: 1
bash (4667): drop_caches: 1
bash (6683): drop_caches: 1
bash (9366): drop_caches: 1
bash (12164): drop_caches: 1
bash (14155): drop_caches: 1
bash (16329): drop_caches: 1
bash (18606): drop_caches: 1
bash (20648): drop_caches: 1
bash (22836): drop_caches: 1
bash (25164): drop_caches: 1
bash (27307): drop_caches: 1
bash (29652): drop_caches: 1
bash (31834): drop_caches: 1
bash (1794): drop_caches: 1
bash (4129): drop_caches: 1
bash (6667): drop_caches: 1
bash (9154): drop_caches: 1
bash (11675): drop_caches: 1
bash (14265): drop_caches: 1
bash (16633): drop_caches: 1
bash (19122): drop_caches: 1
bash (21316): drop_caches: 1
bash (23560): drop_caches: 1
bash (26043): drop_caches: 1
bash (28188): drop_caches: 1
bash (30670): drop_caches: 1
bash (661): drop_caches: 1
bash (2760): drop_caches: 1
bash (5094): drop_caches: 1
bash (7315): drop_caches: 1
bash (10116): drop_caches: 1
bash (12698): drop_caches: 1
bash (15087): drop_caches: 1
bash (17615): drop_caches: 1
bash (20157): drop_caches: 1
bash (22500): drop_caches: 1
bash (24675): drop_caches: 1
bash (26866): drop_caches: 1
bash (29174): drop_caches: 1
bash (31454): drop_caches: 1
bash (1217): drop_caches: 1
bash (3727): drop_caches: 1
bash (6409): drop_caches: 1
bash (8667): drop_caches: 1
bash (11668): drop_caches: 1
bash (14152): drop_caches: 1
bash (16466): drop_caches: 1
bash (18841): drop_caches: 1
bash (21169): drop_caches: 1
bash (23332): drop_caches: 1
bash (25690): drop_caches: 1
bash (27893): drop_caches: 1
bash (30064): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 41.2M 535M 0 197 0 2.40M
lustre-mdt1 43.7M 532M 0 211 0 2.53M
lustre-mdt1 46.1M 530M 0 553 0 4.67M
lustre-mdt1 46.0M 530M 0 165 0 2.07M
lustre-mdt1 46.2M 530M 0 219 0 2.46M
lustre-mdt1 47.6M 528M 0 333 0 4.56M
lustre-mdt1 47.2M 529M 0 602 0 4.27M
lustre-mdt1 47.1M 529M 0 197 0 2.08M
lustre-mdt1 47.0M 529M 0 198 0 2.50M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD ba8f4067 PUD bab82067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk serio_raw i2c_core floppy [last unloaded: libcfs]
CPU: 3 PID: 979 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031bae1280 ti: ffff88031b1d4000 task.ti: ffff88031b1d4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b1d7e50 EFLAGS: 00010246
RAX: ffff88031b1d7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331ad3a98 RDI: 0000000000000063
RBP: ffff88031b1d7e50 R08: 0000000000000000 R09: 000000000000089a
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f773a699740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000b9016000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 52.0M 524M 0 191 0 2.67M
lustre-mdt1 51.8M 524M 0 275 0 3.98M
lustre-mdt1 51.7M 524M 0 19 0 41.5K
lustre-mdt1 51.2M 525M 0 393 0 4.43M
lustre-mdt1 52.0M 524M 0 153 0 2.19M
lustre-mdt1 51.5M 525M 0 157 0 2.08M
lustre-mdt1 52.0M 524M 0 359 0 4.53M
lustre-mdt1 51.8M 524M 0 231 0 1.97M
lustre-mdt1 51.4M 525M 0 224 0 2.14M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b0d6067 PUD 31bf7a067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk floppy serio_raw libata i2c_core [last unloaded: libcfs]
CPU: 9 PID: 999 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b97a4f0 ti: ffff88031fbf4000 task.ti: ffff88031fbf4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fbf7e50 EFLAGS: 00010246
RAX: ffff88031fbf7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031fbf7e50 R08: 0000000000000000 R09: 0000000000006318
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fb1e12fb740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000318cbe000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x240000405 to 0x240000406
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 56.7M 519M 0 512 0 5.88M
lustre-mdt1 55.9M 520M 0 308 0 3.74M
lustre-mdt1 56.6M 519M 0 234 0 3.16M
lustre-mdt1 54.4M 522M 0 429 0 3.70M
lustre-mdt1 54.8M 521M 0 237 0 4.10M
lustre-mdt1 53.7M 522M 0 290 0 3.36M
lustre-mdt1 52.6M 523M 0 34 0 59.9K
lustre-mdt1 52.7M 523M 0 358 0 5.15M
lustre-mdt1 53.2M 523M 0 295 0 4.58M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31edb0067 PUD 31b031067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk libata floppy serio_raw i2c_core [last unloaded: libcfs]
CPU: 12 PID: 1020 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031bf78010 ti: ffff88031af7c000 task.ti: ffff88031af7c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031af7fe50 EFLAGS: 00010246
RAX: ffff88031af7ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d13a98 RDI: 0000000000000063
RBP: ffff88031af7fe50 R08: 0000000000000000 R09: 0000000000003552
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fa29354b740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031981a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.1M 522M 0 518 0 6.04M
lustre-mdt1 54.2M 522M 0 462 0 6.24M
lustre-mdt1 55.1M 521M 0 304 0 3.44M
lustre-mdt1 55.1M 521M 0 344 0 4.13M
lustre-mdt1 54.4M 522M 0 292 0 3.59M
lustre-mdt1 54.5M 521M 0 450 0 6.37M
lustre-mdt1 54.1M 522M 0 284 0 4.05M
lustre-mdt1 55.1M 521M 0 239 0 3.42M
lustre-mdt1 56.7M 519M 0 361 0 4.49M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bac09067 PUD bb8f7067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk libata serio_raw i2c_core [last unloaded: libcfs]
CPU: 0 PID: 989 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031f2fc9d0 ti: ffff8800ba634000 task.ti: ffff8800ba634000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba637e50 EFLAGS: 00010246
RAX: ffff8800ba637fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a13a98 RDI: 0000000000000063
RBP: ffff8800ba637e50 R08: 0000000000000000 R09: 0000000000000a2d
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f3ab1891740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000000ba404000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802c09d8958: disconnect after 22s idle
Lustre: Skipped 3 previous similar messages
bash (24257): drop_caches: 1
bash (26932): drop_caches: 1
bash (28648): drop_caches: 1
bash (31870): drop_caches: 1
bash (1574): drop_caches: 1
bash (3462): drop_caches: 1
bash (6169): drop_caches: 1
bash (7773): drop_caches: 1
bash (10705): drop_caches: 1
bash (12779): drop_caches: 1
bash (15606): drop_caches: 1
bash (17461): drop_caches: 1
bash (20265): drop_caches: 1
bash (21902): drop_caches: 1
bash (24654): drop_caches: 1
bash (26348): drop_caches: 1
bash (28978): drop_caches: 1
bash (30876): drop_caches: 1
bash (1624): drop_caches: 1
bash (3334): drop_caches: 1
bash (5875): drop_caches: 1
bash (7530): drop_caches: 1
bash (10332): drop_caches: 1
bash (12063): drop_caches: 1
bash (14724): drop_caches: 1
bash (16501): drop_caches: 1
bash (19442): drop_caches: 1
bash (21118): drop_caches: 1
bash (23962): drop_caches: 1
bash (25743): drop_caches: 1
bash (28288): drop_caches: 1
bash (30340): drop_caches: 1
Lustre: 30721:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0001: catlog [0x1:0x2:0x0] crosses index zero
bash (1538): drop_caches: 1
bash (3546): drop_caches: 1
bash (6421): drop_caches: 1
bash (8352): drop_caches: 1
bash (11114): drop_caches: 1
bash (13123): drop_caches: 1
bash (15931): drop_caches: 1
bash (17850): drop_caches: 1
bash (20803): drop_caches: 1
bash (22895): drop_caches: 1
bash (26097): drop_caches: 1
bash (27855): drop_caches: 1
bash (31195): drop_caches: 1
bash (979): drop_caches: 1
bash (4273): drop_caches: 1
bash (6064): drop_caches: 1
bash (7912): drop_caches: 1
bash (10858): drop_caches: 1
bash (12619): drop_caches: 1
bash (15481): drop_caches: 1
bash (17499): drop_caches: 1
bash (20196): drop_caches: 1
bash (22129): drop_caches: 1
bash (24901): drop_caches: 1
bash (26639): drop_caches: 1
bash (29478): drop_caches: 1
bash (31629): drop_caches: 1
bash (2368): drop_caches: 1
bash (4402): drop_caches: 1
bash (7472): drop_caches: 1
bash (9450): drop_caches: 1
bash (12568): drop_caches: 1
bash (14598): drop_caches: 1
bash (17569): drop_caches: 1
bash (19521): drop_caches: 1
bash (22621): drop_caches: 1
bash (24580): drop_caches: 1
bash (27423): drop_caches: 1
bash (29375): drop_caches: 1
bash (587): drop_caches: 1
bash (2919): drop_caches: 1
bash (6089): drop_caches: 1
bash (7963): drop_caches: 1
bash (10884): drop_caches: 1
bash (12927): drop_caches: 1
bash (15903): drop_caches: 1
bash (17782): drop_caches: 1
bash (21016): drop_caches: 1
bash (23081): drop_caches: 1
bash (26399): drop_caches: 1
bash (28089): drop_caches: 1
bash (31399): drop_caches: 1
bash (806): drop_caches: 1
bash (3912): drop_caches: 1
bash (5748): drop_caches: 1
bash (8546): drop_caches: 1
bash (10427): drop_caches: 1
bash (13289): drop_caches: 1
bash (15199): drop_caches: 1
bash (17035): drop_caches: 1
bash (19891): drop_caches: 1
bash (21737): drop_caches: 1
bash (24655): drop_caches: 1
bash (26703): drop_caches: 1
bash (29610): drop_caches: 1
Lustre: 30721:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0001: catlog [0x1:0x2:0x0] crosses index zero
bash (31829): drop_caches: 1
bash (2589): drop_caches: 1
bash (4315): drop_caches: 1
bash (6981): drop_caches: 1
bash (8769): drop_caches: 1
bash (11618): drop_caches: 1
bash (13552): drop_caches: 1
bash (16421): drop_caches: 1
bash (18337): drop_caches: 1
bash (21370): drop_caches: 1
bash (23340): drop_caches: 1
bash (26235): drop_caches: 1
bash (27877): drop_caches: 1
bash (30965): drop_caches: 1
bash (684): drop_caches: 1
bash (3767): drop_caches: 1
bash (5509): drop_caches: 1
bash (8250): drop_caches: 1
bash (10139): drop_caches: 1
bash (12963): drop_caches: 1
bash (14878): drop_caches: 1
bash (17634): drop_caches: 1
bash (19316): drop_caches: 1
bash (22003): drop_caches: 1
bash (23833): drop_caches: 1
bash (26798): drop_caches: 1
bash (28463): drop_caches: 1
bash (31650): drop_caches: 1
bash (1393): drop_caches: 1
bash (4355): drop_caches: 1
bash (6277): drop_caches: 1
bash (8897): drop_caches: 1
bash (10532): drop_caches: 1
bash (13164): drop_caches: 1
bash (15083): drop_caches: 1
bash (17647): drop_caches: 1
bash (19296): drop_caches: 1
bash (20908): drop_caches: 1
bash (23573): drop_caches: 1
bash (25437): drop_caches: 1
bash (27931): drop_caches: 1
bash (29770): drop_caches: 1
bash (522): drop_caches: 1
bash (2352): drop_caches: 1
bash (4620): drop_caches: 1
bash (6256): drop_caches: 1
bash (9024): drop_caches: 1
bash (10748): drop_caches: 1
bash (13582): drop_caches: 1
bash (15376): drop_caches: 1
bash (18161): drop_caches: 1
bash (20042): drop_caches: 1
bash (22548): drop_caches: 1
bash (24280): drop_caches: 1
bash (26839): drop_caches: 1
bash (28445): drop_caches: 1
bash (31554): drop_caches: 1
bash (954): drop_caches: 1
bash (4089): drop_caches: 1
bash (5863): drop_caches: 1
bash (8315): drop_caches: 1
bash (10187): drop_caches: 1
bash (12578): drop_caches: 1
bash (14133): drop_caches: 1
bash (16584): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 22.9M 553M 0 0 0 967
lustre-mdt1 22.9M 553M 0 0 0 973
lustre-mdt1 22.9M 553M 0 0 0 983
lustre-mdt1 22.9M 553M 0 0 0 981
lustre-mdt1 22.9M 553M 0 0 0 986
lustre-mdt1 22.9M 553M 0 0 0 970
lustre-mdt1 22.9M 553M 0 0 0 984
lustre-mdt1 22.9M 553M 0 0 0 971
lustre-mdt1 22.9M 553M 0 0 0 975
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b9b1067 PUD 319231067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 7 PID: 981 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800b94e9280 ti: ffff88031b308000 task.ti: ffff88031b308000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b30be50 EFLAGS: 00010246
RAX: ffff88031b30bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331bd3a98 RDI: 0000000000000063
RBP: ffff88031b30be50 R08: 0000000000000000 R09: 0000000000002459
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fce1cfa6740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000031dfd6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 25.0M 551M 0 0 0 980
lustre-mdt1 25.0M 551M 0 0 0 863
lustre-mdt1 25.0M 551M 0 0 0 983
lustre-mdt1 25.0M 551M 0 0 0 980
lustre-mdt1 25.0M 551M 0 0 0 956
lustre-mdt1 25.0M 551M 0 0 0 946
lustre-mdt1 25.0M 551M 0 0 0 942
lustre-mdt1 25.0M 551M 0 0 0 980
lustre-mdt1 25.0M 551M 0 0 0 891
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a6c5067 PUD 31b233067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw libata i2c_core virtio_blk floppy [last unloaded: libcfs]
CPU: 3 PID: 1013 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880319ee3760 ti: ffff88031fd0c000 task.ti: ffff88031fd0c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fd0fe50 EFLAGS: 00010246
RAX: ffff88031fd0ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331ad3a98 RDI: 0000000000000063
RBP: ffff88031fd0fe50 R08: 0000000000000000 R09: 00000000000024cd
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f5d335c1740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031cdce000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8800a09fb7e8: disconnect after 24s idle
bash (23347): drop_caches: 1
bash (23958): drop_caches: 1
bash (24399): drop_caches: 1
bash (24869): drop_caches: 1
bash (25354): drop_caches: 1
bash (25815): drop_caches: 1
bash (26291): drop_caches: 1
bash (26878): drop_caches: 1
bash (27403): drop_caches: 1
bash (27972): drop_caches: 1
bash (28483): drop_caches: 1
bash (29060): drop_caches: 1
bash (29629): drop_caches: 1
bash (30162): drop_caches: 1
bash (30695): drop_caches: 1
bash (31139): drop_caches: 1
bash (31681): drop_caches: 1
bash (32092): drop_caches: 1
bash (32688): drop_caches: 1
bash (651): drop_caches: 1
bash (1095): drop_caches: 1
bash (1539): drop_caches: 1
bash (2090): drop_caches: 1
bash (2705): drop_caches: 1
bash (3168): drop_caches: 1
bash (3809): drop_caches: 1
bash (4293): drop_caches: 1
bash (4967): drop_caches: 1
bash (5584): drop_caches: 1
bash (6252): drop_caches: 1
bash (6675): drop_caches: 1
bash (8517): drop_caches: 1
bash (10016): drop_caches: 1
bash (11674): drop_caches: 1
bash (13251): drop_caches: 1
bash (14800): drop_caches: 1
bash (16512): drop_caches: 1
bash (18181): drop_caches: 1
bash (19853): drop_caches: 1
bash (21456): drop_caches: 1
bash (23037): drop_caches: 1
bash (24576): drop_caches: 1
bash (26183): drop_caches: 1
bash (27677): drop_caches: 1
bash (29150): drop_caches: 1
bash (30699): drop_caches: 1
bash (32454): drop_caches: 1
bash (1864): drop_caches: 1
bash (3411): drop_caches: 1
bash (5147): drop_caches: 1
bash (6855): drop_caches: 1
bash (8475): drop_caches: 1
bash (10319): drop_caches: 1
bash (11869): drop_caches: 1
bash (13438): drop_caches: 1
bash (14909): drop_caches: 1
bash (16534): drop_caches: 1
bash (18147): drop_caches: 1
bash (19783): drop_caches: 1
bash (21303): drop_caches: 1
bash (22761): drop_caches: 1
bash (24577): drop_caches: 1
bash (26228): drop_caches: 1
bash (27795): drop_caches: 1
bash (29353): drop_caches: 1
bash (30954): drop_caches: 1
bash (32490): drop_caches: 1
bash (1662): drop_caches: 1
bash (3209): drop_caches: 1
bash (4956): drop_caches: 1
bash (6963): drop_caches: 1
bash (8664): drop_caches: 1
bash (10508): drop_caches: 1
bash (12157): drop_caches: 1
bash (13848): drop_caches: 1
bash (15286): drop_caches: 1
bash (16770): drop_caches: 1
bash (18319): drop_caches: 1
bash (19701): drop_caches: 1
bash (21277): drop_caches: 1
bash (22854): drop_caches: 1
bash (24500): drop_caches: 1
bash (26094): drop_caches: 1
bash (27713): drop_caches: 1
bash (29265): drop_caches: 1
bash (30884): drop_caches: 1
bash (32405): drop_caches: 1
bash (1534): drop_caches: 1
bash (3143): drop_caches: 1
bash (4871): drop_caches: 1
bash (6844): drop_caches: 1
bash (8631): drop_caches: 1
bash (10275): drop_caches: 1
bash (12047): drop_caches: 1
bash (13599): drop_caches: 1
bash (15221): drop_caches: 1
bash (16872): drop_caches: 1
bash (18583): drop_caches: 1
bash (20129): drop_caches: 1
bash (21752): drop_caches: 1
bash (23354): drop_caches: 1
bash (25031): drop_caches: 1
bash (26590): drop_caches: 1
bash (28241): drop_caches: 1
bash (29976): drop_caches: 1
bash (31606): drop_caches: 1
bash (952): drop_caches: 1
bash (2531): drop_caches: 1
bash (4184): drop_caches: 1
bash (6137): drop_caches: 1
bash (7873): drop_caches: 1
bash (9657): drop_caches: 1
bash (11322): drop_caches: 1
bash (12834): drop_caches: 1
bash (14547): drop_caches: 1
bash (16274): drop_caches: 1
bash (17921): drop_caches: 1
bash (19504): drop_caches: 1
bash (21157): drop_caches: 1
bash (22759): drop_caches: 1
bash (24446): drop_caches: 1
bash (26089): drop_caches: 1
bash (27677): drop_caches: 1
bash (29503): drop_caches: 1
bash (31036): drop_caches: 1
bash (32417): drop_caches: 1
bash (1650): drop_caches: 1
bash (3326): drop_caches: 1
bash (5102): drop_caches: 1
bash (6895): drop_caches: 1
bash (8445): drop_caches: 1
bash (10127): drop_caches: 1
bash (11601): drop_caches: 1
bash (13014): drop_caches: 1
bash (14284): drop_caches: 1
bash (15756): drop_caches: 1
bash (17352): drop_caches: 1
bash (18800): drop_caches: 1
bash (20435): drop_caches: 1
bash (21948): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 29.4M 547M 0 0 0 983
lustre-mdt1 29.6M 546M 0 66 0 282K
lustre-mdt1 29.6M 546M 0 0 0 977
lustre-mdt1 29.6M 546M 0 0 0 981
lustre-mdt1 29.6M 546M 0 0 0 980
lustre-mdt1 29.6M 546M 0 0 0 984
lustre-mdt1 27.2M 549M 0 0 0 979
bash (23526): drop_caches: 1
lustre-mdt1 27.2M 549M 0 0 0 983
lustre-mdt1 27.2M 549M 0 0 0 979
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31aa14067 PUD 31a923067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr i2c_piix4 virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks i2c_core ata_piix floppy libata serio_raw virtio_blk [last unloaded: libcfs]
CPU: 12 PID: 1009 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800ba023760 ti: ffff88031f240000 task.ti: ffff88031f240000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031f243e50 EFLAGS: 00010246
RAX: ffff88031f243fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031f243e50 R08: 0000000000000000 R09: 0000000000000949
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f30dc3da740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031d0ba000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8800af2437e8: disconnect after 22s idle
Lustre: Skipped 3 previous similar messages
bash (23546): drop_caches: 1
bash (25535): drop_caches: 1
bash (28227): drop_caches: 1
bash (30332): drop_caches: 1
bash (624): drop_caches: 1
bash (2641): drop_caches: 1
bash (5371): drop_caches: 1
bash (7231): drop_caches: 1
bash (9959): drop_caches: 1
bash (12059): drop_caches: 1
bash (14769): drop_caches: 1
bash (16794): drop_caches: 1
bash (20219): drop_caches: 1
bash (22709): drop_caches: 1
bash (25272): drop_caches: 1
bash (27373): drop_caches: 1
bash (30437): drop_caches: 1
bash (32653): drop_caches: 1
bash (3314): drop_caches: 1
bash (5347): drop_caches: 1
bash (8322): drop_caches: 1
bash (10570): drop_caches: 1
bash (13806): drop_caches: 1
bash (16058): drop_caches: 1
bash (19302): drop_caches: 1
bash (22034): drop_caches: 1
bash (25287): drop_caches: 1
bash (27383): drop_caches: 1
bash (30204): drop_caches: 1
bash (32445): drop_caches: 1
bash (3275): drop_caches: 1
Lustre: 20022:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (5513): drop_caches: 1
bash (8425): drop_caches: 1
bash (10408): drop_caches: 1
bash (13358): drop_caches: 1
bash (15307): drop_caches: 1
bash (17531): drop_caches: 1
bash (20781): drop_caches: 1
bash (23442): drop_caches: 1
bash (26361): drop_caches: 1
bash (28589): drop_caches: 1
bash (31166): drop_caches: 1
bash (832): drop_caches: 1
bash (3788): drop_caches: 1
bash (5800): drop_caches: 1
bash (8596): drop_caches: 1
bash (10745): drop_caches: 1
bash (13251): drop_caches: 1
bash (15345): drop_caches: 1
bash (18349): drop_caches: 1
bash (21005): drop_caches: 1
bash (24393): drop_caches: 1
bash (26520): drop_caches: 1
bash (29257): drop_caches: 1
bash (31446): drop_caches: 1
bash (1629): drop_caches: 1
bash (3911): drop_caches: 1
bash (6745): drop_caches: 1
bash (8843): drop_caches: 1
bash (11904): drop_caches: 1
bash (14019): drop_caches: 1
bash (17319): drop_caches: 1
bash (19538): drop_caches: 1
bash (23133): drop_caches: 1
bash (25211): drop_caches: 1
bash (27865): drop_caches: 1
bash (30152): drop_caches: 1
bash (314): drop_caches: 1
bash (2532): drop_caches: 1
bash (5470): drop_caches: 1
bash (7517): drop_caches: 1
bash (10625): drop_caches: 1
bash (12779): drop_caches: 1
bash (15609): drop_caches: 1
bash (17694): drop_caches: 1
bash (20767): drop_caches: 1
bash (23147): drop_caches: 1
bash (25824): drop_caches: 1
bash (27592): drop_caches: 1
bash (29609): drop_caches: 1
bash (32203): drop_caches: 1
bash (1998): drop_caches: 1
bash (4628): drop_caches: 1
bash (6713): drop_caches: 1
bash (9108): drop_caches: 1
bash (10862): drop_caches: 1
bash (13294): drop_caches: 1
bash (15298): drop_caches: 1
bash (17889): drop_caches: 1
bash (19982): drop_caches: 1
bash (22903): drop_caches: 1
bash (24783): drop_caches: 1
bash (27325): drop_caches: 1
bash (29428): drop_caches: 1
bash (32123): drop_caches: 1
bash (1737): drop_caches: 1
bash (4380): drop_caches: 1
bash (6520): drop_caches: 1
bash (9045): drop_caches: 1
bash (11015): drop_caches: 1
bash (13552): drop_caches: 1
bash (15316): drop_caches: 1
bash (17978): drop_caches: 1
bash (20566): drop_caches: 1
bash (23658): drop_caches: 1
bash (25761): drop_caches: 1
bash (28362): drop_caches: 1
bash (30342): drop_caches: 1
bash (525): drop_caches: 1
bash (2605): drop_caches: 1
bash (5753): drop_caches: 1
bash (8105): drop_caches: 1
bash (11180): drop_caches: 1
bash (13208): drop_caches: 1
bash (16234): drop_caches: 1
bash (18671): drop_caches: 1
bash (22198): drop_caches: 1
bash (23956): drop_caches: 1
bash (26867): drop_caches: 1
bash (28915): drop_caches: 1
bash (31713): drop_caches: 1
bash (1481): drop_caches: 1
bash (3379): drop_caches: 1
bash (6441): drop_caches: 1
bash (8296): drop_caches: 1
bash (10858): drop_caches: 1
bash (12773): drop_caches: 1
bash (15353): drop_caches: 1
bash (17427): drop_caches: 1
bash (20288): drop_caches: 1
bash (22760): drop_caches: 1
bash (25355): drop_caches: 1
bash (27393): drop_caches: 1
bash (29849): drop_caches: 1
bash (31868): drop_caches: 1
bash (2273): drop_caches: 1
bash (4027): drop_caches: 1
bash (6806): drop_caches: 1
bash (8681): drop_caches: 1
bash (11697): drop_caches: 1
bash (13864): drop_caches: 1
bash (16911): drop_caches: 1
bash (18910): drop_caches: 1
bash (22204): drop_caches: 1
bash (24229): drop_caches: 1
bash (27270): drop_caches: 1
bash (29303): drop_caches: 1
bash (32324): drop_caches: 1
bash (2081): drop_caches: 1
bash (4754): drop_caches: 1
bash (7087): drop_caches: 1
bash (9917): drop_caches: 1
bash (11703): drop_caches: 1
bash (14776): drop_caches: 1
bash (16792): drop_caches: 1
bash (19621): drop_caches: 1
bash (22176): drop_caches: 1
bash (25025): drop_caches: 1
bash (27092): drop_caches: 1
bash (30030): drop_caches: 1
Lustre: 20022:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (31953): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 47.1M 529M 0 517 0 4.37M
lustre-mdt1 41.8M 534M 0 63 0 329K
lustre-mdt1 41.8M 534M 0 0 0 973
lustre-mdt1 41.8M 534M 0 0 0 997
lustre-mdt1 41.8M 534M 0 0 0 998
lustre-mdt1 42.4M 534M 0 81 0 731K
lustre-mdt1 42.4M 534M 0 0 0 994
lustre-mdt1 42.4M 534M 0 0 0 988
lustre-mdt1 42.4M 534M 0 0 0 985
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31bb3f067 PUD 31b160067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs]
CPU: 0 PID: 1006 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ed249d0 ti: ffff8800b96c8000 task.ti: ffff8800b96c8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800b96cbe50 EFLAGS: 00010246
RAX: ffff8800b96cbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a13a98 RDI: 0000000000000063
RBP: ffff8800b96cbe50 R08: 0000000000000000 R09: 00000000000023a8
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f928f65a740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031aa02000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 23.0M 553M 0 0 0 951
lustre-mdt1 23.0M 553M 0 0 0 905
lustre-mdt1 23.0M 553M 0 1 0 1.94K
lustre-mdt1 23.0M 553M 0 0 0 979
lustre-mdt1 23.0M 553M 0 0 0 971
lustre-mdt1 23.0M 553M 0 0 0 968
lustre-mdt1 23.0M 553M 0 0 0 941
lustre-mdt1 23.0M 553M 0 0 0 874
lustre-mdt1 23.0M 553M 0 0 0 793
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 319adc067 PUD 31a218067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw i2c_core virtio_blk libata [last unloaded: libcfs]
CPU: 10 PID: 990 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800bae349d0 ti: ffff8800ba954000 task.ti: ffff8800ba954000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800ba957e50 EFLAGS: 00010246
RAX: ffff8800ba957fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800ba957e50 R08: 0000000000000000 R09: 0000000000001fa7
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f327439c740(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 0000000000000000 CR3: 000000031993e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 51.7M 524M 0 211 0 2.91M
lustre-mdt1 52.3M 524M 0 355 0 4.74M
lustre-mdt1 52.3M 524M 0 324 0 4.06M
lustre-mdt1 53.4M 523M 0 214 0 2.98M
lustre-mdt1 52.1M 524M 0 398 0 4.67M
lustre-mdt1 51.7M 524M 0 340 0 4.13M
lustre-mdt1 51.8M 524M 0 328 0 4.44M
lustre-mdt1 53.6M 522M 0 193 0 3.18M
lustre-mdt1 52.6M 523M 0 148 0 1.66M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3194d2067 PUD 31bf05067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw virtio_blk floppy i2c_core [last unloaded: libcfs]
CPU: 7 PID: 1010 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800baba24f0 ti: ffff8800bacf4000 task.ti: ffff8800bacf4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bacf7e50 EFLAGS: 00010246
RAX: ffff8800bacf7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800bacf7e50 R08: 0000000000000000 R09: 000000000000eaa4
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f28ba7be740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031a898000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8801887fdd28: disconnect after 23s idle
Lustre: Skipped 3 previous similar messages
bash (29974): drop_caches: 1
bash (31906): drop_caches: 1
bash (2195): drop_caches: 1
bash (4310): drop_caches: 1
bash (7047): drop_caches: 1
bash (8935): drop_caches: 1
bash (11749): drop_caches: 1
bash (13687): drop_caches: 1
bash (16195): drop_caches: 1
bash (18324): drop_caches: 1
bash (21513): drop_caches: 1
bash (23227): drop_caches: 1
bash (25822): drop_caches: 1
bash (27740): drop_caches: 1
bash (30382): drop_caches: 1
bash (32213): drop_caches: 1
bash (1747): drop_caches: 1
bash (4603): drop_caches: 1
bash (6395): drop_caches: 1
bash (8823): drop_caches: 1
bash (10798): drop_caches: 1
bash (13354): drop_caches: 1
bash (15341): drop_caches: 1
bash (18118): drop_caches: 1
bash (20696): drop_caches: 1
bash (23532): drop_caches: 1
bash (25512): drop_caches: 1
bash (28465): drop_caches: 1
bash (30318): drop_caches: 1
bash (596): drop_caches: 1
bash (2509): drop_caches: 1
Lustre: 18850:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (5195): drop_caches: 1
bash (7141): drop_caches: 1
bash (9920): drop_caches: 1
bash (11573): drop_caches: 1
bash (14133): drop_caches: 1
bash (16049): drop_caches: 1
bash (18524): drop_caches: 1
bash (20735): drop_caches: 1
bash (23480): drop_caches: 1
bash (25482): drop_caches: 1
bash (28232): drop_caches: 1
bash (30200): drop_caches: 1
bash (32571): drop_caches: 1
bash (1969): drop_caches: 1
bash (4313): drop_caches: 1
bash (6196): drop_caches: 1
bash (8953): drop_caches: 1
bash (10775): drop_caches: 1
bash (13234): drop_caches: 1
bash (15164): drop_caches: 1
bash (17808): drop_caches: 1
bash (20047): drop_caches: 1
Lustre: lustre-OST0000: already connected client lustre-MDT0000-mdtlov_UUID (at 0@lo) with handle 0xa9bf061e2f4104a6. Rejecting client with the same UUID trying to reconnect with handle 0x1d16958a39a30cab
bash (23337): drop_caches: 1
bash (25216): drop_caches: 1
bash (27913): drop_caches: 1
bash (29879): drop_caches: 1
bash (32635): drop_caches: 1
bash (2234): drop_caches: 1
bash (4212): drop_caches: 1
bash (6964): drop_caches: 1
bash (8936): drop_caches: 1
bash (11613): drop_caches: 1
bash (13537): drop_caches: 1
bash (16085): drop_caches: 1
bash (18233): drop_caches: 1
bash (21756): drop_caches: 1
bash (23581): drop_caches: 1
bash (26341): drop_caches: 1
bash (28519): drop_caches: 1
bash (31257): drop_caches: 1
bash (812): drop_caches: 1
bash (4142): drop_caches: 1
bash (6296): drop_caches: 1
bash (9448): drop_caches: 1
bash (11382): drop_caches: 1
bash (14346): drop_caches: 1
bash (16559): drop_caches: 1
bash (19830): drop_caches: 1
bash (22110): drop_caches: 1
bash (25153): drop_caches: 1
bash (27145): drop_caches: 1
bash (30295): drop_caches: 1
bash (32395): drop_caches: 1
bash (2778): drop_caches: 1
bash (4876): drop_caches: 1
bash (7754): drop_caches: 1
bash (9899): drop_caches: 1
bash (12791): drop_caches: 1
bash (14741): drop_caches: 1
bash (17375): drop_caches: 1
bash (19576): drop_caches: 1
bash (22589): drop_caches: 1
bash (24481): drop_caches: 1
bash (27500): drop_caches: 1
Lustre: 18850:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (29613): drop_caches: 1
bash (32657): drop_caches: 1
bash (2309): drop_caches: 1
bash (5327): drop_caches: 1
bash (7323): drop_caches: 1
bash (10517): drop_caches: 1
bash (12750): drop_caches: 1
bash (14889): drop_caches: 1
bash (17392): drop_caches: 1
bash (19599): drop_caches: 1
bash (22482): drop_caches: 1
bash (24527): drop_caches: 1
bash (27350): drop_caches: 1
bash (29381): drop_caches: 1
bash (32094): drop_caches: 1
bash (1668): drop_caches: 1
bash (4396): drop_caches: 1
bash (6354): drop_caches: 1
bash (9152): drop_caches: 1
bash (11240): drop_caches: 1
bash (13977): drop_caches: 1
bash (16152): drop_caches: 1
bash (19010): drop_caches: 1
bash (21283): drop_caches: 1
bash (24160): drop_caches: 1
bash (26266): drop_caches: 1
bash (29228): drop_caches: 1
bash (31174): drop_caches: 1
bash (1377): drop_caches: 1
bash (3463): drop_caches: 1
bash (6159): drop_caches: 1
bash (8229): drop_caches: 1
bash (10702): drop_caches: 1
bash (12744): drop_caches: 1
bash (15597): drop_caches: 1
bash (17643): drop_caches: 1
bash (20794): drop_caches: 1
bash (23011): drop_caches: 1
bash (25883): drop_caches: 1
bash (27930): drop_caches: 1
bash (30670): drop_caches: 1
bash (32731): drop_caches: 1
bash (2977): drop_caches: 1
bash (5069): drop_caches: 1
bash (7966): drop_caches: 1
bash (10168): drop_caches: 1
bash (13189): drop_caches: 1
bash (15166): drop_caches: 1
bash (18115): drop_caches: 1
bash (20673): drop_caches: 1
bash (22825): drop_caches: 1
bash (25624): drop_caches: 1
bash (27654): drop_caches: 1
bash (30421): drop_caches: 1
bash (32499): drop_caches: 1
bash (2786): drop_caches: 1
bash (5002): drop_caches: 1
bash (7777): drop_caches: 1
bash (9894): drop_caches: 1
bash (12657): drop_caches: 1
bash (14726): drop_caches: 1
bash (17511): drop_caches: 1
bash (19607): drop_caches: 1
bash (22596): drop_caches: 1
bash (24746): drop_caches: 1
bash (27539): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 45.1M 531M 0 0 0 989
lustre-mdt1 45.1M 531M 0 0 0 995
lustre-mdt1 45.1M 531M 0 0 0 989
lustre-mdt1 43.2M 533M 0 95 0 718K
lustre-mdt1 43.2M 533M 0 0 0 992
lustre-mdt1 43.2M 533M 0 0 0 992
lustre-mdt1 43.2M 533M 0 0 0 996
lustre-mdt1 43.2M 533M 0 0 0 993
lustre-mdt1 40.6M 535M 0 0 0 987
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31d91a067 PUD 31d91b067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata floppy i2c_core [last unloaded: libcfs]
CPU: 7 PID: 1000 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031fd53760 ti: ffff88031e3d0000 task.ti: ffff88031e3d0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031e3d3e50 EFLAGS: 00010246
RAX: ffff88031e3d3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031e3d3e50 R08: 0000000000000000 R09: 000000000000632e
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fc97cbde740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319274000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff880022fb9bf8: disconnect after 24s idle
Lustre: Skipped 3 previous similar messages
bash (20939): drop_caches: 1
bash (24073): drop_caches: 1
bash (26136): drop_caches: 1
bash (29350): drop_caches: 1
bash (31498): drop_caches: 1
bash (1996): drop_caches: 1
bash (3991): drop_caches: 1
bash (6012): drop_caches: 1
bash (8940): drop_caches: 1
bash (11111): drop_caches: 1
bash (14552): drop_caches: 1
bash (17040): drop_caches: 1
bash (20132): drop_caches: 1
bash (22317): drop_caches: 1
bash (25176): drop_caches: 1
bash (27543): drop_caches: 1
bash (30564): drop_caches: 1
bash (32624): drop_caches: 1
bash (3160): drop_caches: 1
bash (5333): drop_caches: 1
bash (8171): drop_caches: 1
bash (10427): drop_caches: 1
bash (13525): drop_caches: 1
bash (16077): drop_caches: 1
bash (19045): drop_caches: 1
bash (21150): drop_caches: 1
bash (24187): drop_caches: 1
bash (26469): drop_caches: 1
bash (29402): drop_caches: 1
bash (31733): drop_caches: 1
bash (2194): drop_caches: 1
bash (4157): drop_caches: 1
bash (7171): drop_caches: 1
bash (9259): drop_caches: 1
bash (12283): drop_caches: 1
bash (14772): drop_caches: 1
bash (18082): drop_caches: 1
bash (19979): drop_caches: 1
bash (22700): drop_caches: 1
bash (24767): drop_caches: 1
bash (27693): drop_caches: 1
bash (29794): drop_caches: 1
bash (602): drop_caches: 1
bash (2792): drop_caches: 1
bash (5937): drop_caches: 1
bash (8048): drop_caches: 1
bash (10837): drop_caches: 1
bash (13041): drop_caches: 1
bash (16461): drop_caches: 1
bash (18468): drop_caches: 1
bash (20521): drop_caches: 1
bash (23416): drop_caches: 1
bash (25366): drop_caches: 1
bash (27752): drop_caches: 1
bash (29624): drop_caches: 1
bash (32413): drop_caches: 1
bash (2089): drop_caches: 1
bash (4688): drop_caches: 1
bash (6737): drop_caches: 1
bash (9497): drop_caches: 1
bash (11469): drop_caches: 1
bash (14207): drop_caches: 1
bash (16617): drop_caches: 1
bash (19436): drop_caches: 1
bash (21589): drop_caches: 1
bash (24398): drop_caches: 1
bash (26330): drop_caches: 1
bash (29275): drop_caches: 1
bash (31188): drop_caches: 1
bash (1502): drop_caches: 1
bash (3473): drop_caches: 1
bash (6208): drop_caches: 1
bash (8315): drop_caches: 1
bash (11067): drop_caches: 1
bash (12977): drop_caches: 1
bash (16098): drop_caches: 1
bash (18279): drop_caches: 1
bash (21057): drop_caches: 1
bash (23205): drop_caches: 1
bash (26390): drop_caches: 1
bash (28233): drop_caches: 1
bash (30997): drop_caches: 1
bash (624): drop_caches: 1
bash (3837): drop_caches: 1
bash (6009): drop_caches: 1
bash (8913): drop_caches: 1
bash (10963): drop_caches: 1
bash (13604): drop_caches: 1
bash (16082): drop_caches: 1
bash (18674): drop_caches: 1
bash (20659): drop_caches: 1
bash (22824): drop_caches: 1
bash (25473): drop_caches: 1
bash (27382): drop_caches: 1
bash (29915): drop_caches: 1
bash (31678): drop_caches: 1
bash (1848): drop_caches: 1
bash (3600): drop_caches: 1
bash (5948): drop_caches: 1
bash (8037): drop_caches: 1
bash (10999): drop_caches: 1
bash (13117): drop_caches: 1
bash (16566): drop_caches: 1
bash (18810): drop_caches: 1
bash (21475): drop_caches: 1
bash (23680): drop_caches: 1
bash (26711): drop_caches: 1
bash (28629): drop_caches: 1
bash (31569): drop_caches: 1
bash (1120): drop_caches: 1
bash (4027): drop_caches: 1
bash (5876): drop_caches: 1
bash (8826): drop_caches: 1
bash (10830): drop_caches: 1
bash (13393): drop_caches: 1
bash (15480): drop_caches: 1
bash (18585): drop_caches: 1
bash (20409): drop_caches: 1
bash (23166): drop_caches: 1
bash (25088): drop_caches: 1
bash (27797): drop_caches: 1
bash (29733): drop_caches: 1
bash (32755): drop_caches: 1
bash (2449): drop_caches: 1
bash (5505): drop_caches: 1
bash (7432): drop_caches: 1
bash (10269): drop_caches: 1
bash (12311): drop_caches: 1
bash (15670): drop_caches: 1
bash (17844): drop_caches: 1
bash (20392): drop_caches: 1
bash (22483): drop_caches: 1
bash (25116): drop_caches: 1
bash (27203): drop_caches: 1
bash (29142): drop_caches: 1
bash (31947): drop_caches: 1
bash (1538): drop_caches: 1
bash (4309): drop_caches: 1
bash (6357): drop_caches: 1
bash (9203): drop_caches: 1
bash (11363): drop_caches: 1
bash (14388): drop_caches: 1
bash (16679): drop_caches: 1
bash (19409): drop_caches: 1
bash (21340): drop_caches: 1
bash (24060): drop_caches: 1
bash (26004): drop_caches: 1
bash (28729): drop_caches: 1
bash (30668): drop_caches: 1
bash (555): drop_caches: 1
bash (2421): drop_caches: 1
bash (5151): drop_caches: 1
bash (7097): drop_caches: 1
bash (9501): drop_caches: 1
bash (11577): drop_caches: 1
bash (14571): drop_caches: 1
bash (16647): drop_caches: 1
bash (19362): drop_caches: 1
bash (21573): drop_caches: 1
bash (24696): drop_caches: 1
bash (26756): drop_caches: 1
bash (29697): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.0M 537M 0 5 0 18.0K
lustre-mdt1 39.1M 537M 0 72 0 320K
lustre-mdt1 39.1M 537M 0 0 0 989
lustre-mdt1 39.1M 537M 0 0 0 991
lustre-mdt1 39.1M 537M 0 0 0 995
lustre-mdt1 38.6M 537M 0 71 0 208K
lustre-mdt1 38.6M 537M 0 0 0 990
lustre-mdt1 38.6M 537M 0 0 0 993
lustre-mdt1 38.6M 537M 0 0 0 990
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e7f2067 PUD 31e67a067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy virtio_blk libata i2c_core [last unloaded: libcfs]
CPU: 2 PID: 1044 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ec68010 ti: ffff88031dc40000 task.ti: ffff88031dc40000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031dc43e50 EFLAGS: 00010246
RAX: ffff88031dc43fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a93a98 RDI: 0000000000000063
RBP: ffff88031dc43e50 R08: 0000000000000000 R09: 000000000000901a
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fc29e6ba740(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031fb56000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802ed06c138: disconnect after 21s idle
Lustre: Skipped 2 previous similar messages
bash (8636): drop_caches: 1
bash (11886): drop_caches: 1
bash (14233): drop_caches: 1
bash (16831): drop_caches: 1
bash (18888): drop_caches: 1
bash (21493): drop_caches: 1
bash (23459): drop_caches: 1
bash (26087): drop_caches: 1
bash (28082): drop_caches: 1
bash (31128): drop_caches: 1
bash (429): drop_caches: 1
bash (2826): drop_caches: 1
bash (5004): drop_caches: 1
bash (8237): drop_caches: 1
bash (10344): drop_caches: 1
bash (13442): drop_caches: 1
bash (15480): drop_caches: 1
bash (18396): drop_caches: 1
bash (20430): drop_caches: 1
bash (23436): drop_caches: 1
bash (25477): drop_caches: 1
bash (28096): drop_caches: 1
bash (30093): drop_caches: 1
bash (492): drop_caches: 1
bash (2543): drop_caches: 1
bash (4965): drop_caches: 1
bash (6851): drop_caches: 1
bash (9730): drop_caches: 1
bash (11923): drop_caches: 1
bash (15003): drop_caches: 1
bash (17078): drop_caches: 1
Lustre: 11653:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (19561): drop_caches: 1
bash (21543): drop_caches: 1
bash (24128): drop_caches: 1
bash (25989): drop_caches: 1
bash (27946): drop_caches: 1
bash (30334): drop_caches: 1
bash (32331): drop_caches: 1
bash (2670): drop_caches: 1
bash (4592): drop_caches: 1
bash (7163): drop_caches: 1
bash (9139): drop_caches: 1
bash (11871): drop_caches: 1
bash (14425): drop_caches: 1
bash (17193): drop_caches: 1
bash (19328): drop_caches: 1
bash (22063): drop_caches: 1
bash (24018): drop_caches: 1
bash (26623): drop_caches: 1
bash (28628): drop_caches: 1
bash (31264): drop_caches: 1
bash (704): drop_caches: 1
bash (3568): drop_caches: 1
bash (5811): drop_caches: 1
bash (8480): drop_caches: 1
bash (10292): drop_caches: 1
bash (13955): drop_caches: 1
bash (15911): drop_caches: 1
bash (18615): drop_caches: 1
bash (20553): drop_caches: 1
bash (22928): drop_caches: 1
bash (24992): drop_caches: 1
bash (27599): drop_caches: 1
bash (29768): drop_caches: 1
bash (32374): drop_caches: 1
bash (1978): drop_caches: 1
bash (4854): drop_caches: 1
bash (7105): drop_caches: 1
bash (9825): drop_caches: 1
bash (11818): drop_caches: 1
bash (14861): drop_caches: 1
bash (16923): drop_caches: 1
bash (19544): drop_caches: 1
bash (21449): drop_caches: 1
bash (24263): drop_caches: 1
bash (26181): drop_caches: 1
bash (28821): drop_caches: 1
bash (30775): drop_caches: 1
bash (32691): drop_caches: 1
bash (3112): drop_caches: 1
bash (5165): drop_caches: 1
bash (8172): drop_caches: 1
bash (10081): drop_caches: 1
bash (13183): drop_caches: 1
bash (15410): drop_caches: 1
bash (18168): drop_caches: 1
bash (20377): drop_caches: 1
bash (23365): drop_caches: 1
bash (25563): drop_caches: 1
bash (28300): drop_caches: 1
bash (30642): drop_caches: 1
bash (1230): drop_caches: 1
bash (3293): drop_caches: 1
bash (6261): drop_caches: 1
bash (8415): drop_caches: 1
bash (11254): drop_caches: 1
bash (13651): drop_caches: 1
bash (16814): drop_caches: 1
bash (18983): drop_caches: 1
bash (21740): drop_caches: 1
bash (23815): drop_caches: 1
bash (26748): drop_caches: 1
bash (29005): drop_caches: 1
bash (32085): drop_caches: 1
bash (1949): drop_caches: 1
bash (4884): drop_caches: 1
bash (6777): drop_caches: 1
bash (9231): drop_caches: 1
bash (11507): drop_caches: 1
bash (15112): drop_caches: 1
bash (17265): drop_caches: 1
bash (20004): drop_caches: 1
bash (22034): drop_caches: 1
bash (24854): drop_caches: 1
bash (27094): drop_caches: 1
bash (30103): drop_caches: 1
bash (32339): drop_caches: 1
bash (2939): drop_caches: 1
bash (5123): drop_caches: 1
bash (8070): drop_caches: 1
bash (10067): drop_caches: 1
bash (12433): drop_caches: 1
bash (15626): drop_caches: 1
bash (17779): drop_caches: 1
bash (20558): drop_caches: 1
bash (22709): drop_caches: 1
bash (25657): drop_caches: 1
bash (27863): drop_caches: 1
bash (30787): drop_caches: 1
bash (659): drop_caches: 1
bash (3821): drop_caches: 1
bash (5748): drop_caches: 1
bash (8599): drop_caches: 1
bash (10885): drop_caches: 1
bash (14489): drop_caches: 1
bash (16677): drop_caches: 1
bash (19730): drop_caches: 1
bash (22006): drop_caches: 1
bash (25062): drop_caches: 1
bash (27091): drop_caches: 1
bash (30260): drop_caches: 1
bash (32428): drop_caches: 1
bash (3048): drop_caches: 1
bash (4951): drop_caches: 1
bash (7923): drop_caches: 1
bash (9942): drop_caches: 1
bash (13066): drop_caches: 1
bash (15394): drop_caches: 1
bash (18454): drop_caches: 1
bash (20452): drop_caches: 1
bash (23043): drop_caches: 1
bash (25487): drop_caches: 1
bash (28381): drop_caches: 1
bash (30375): drop_caches: 1
bash (913): drop_caches: 1
bash (3104): drop_caches: 1
bash (6295): drop_caches: 1
bash (8708): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 43.6M 532M 0 0 0 993
lustre-mdt1 43.6M 532M 0 0 0 999
lustre-mdt1 43.6M 532M 0 0 0 994
lustre-mdt1 43.6M 532M 0 0 0 994
lustre-mdt1 40.9M 535M 0 0 0 995
bash (11816): drop_caches: 1
lustre-mdt1 40.9M 535M 0 0 0 997
lustre-mdt1 41.1M 535M 0 358 0 4.99M
lustre-mdt1 45.4M 531M 0 252 0 2.69M
lustre-mdt1 47.2M 529M 0 295 0 4.13M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 327985067 PUD 31a182067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 14 PID: 1002 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e59dc40 ti: ffff88031dc2c000 task.ti: ffff88031dc2c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031dc2fe50 EFLAGS: 00010246
RAX: ffff88031dc2ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031dc2fe50 R08: 0000000000000000 R09: 0000000000003652
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f10f9b5d740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b9fc000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802a8bc1bf8: disconnect after 22s idle
Lustre: Skipped 2 previous similar messages
bash (6445): drop_caches: 1
bash (8452): drop_caches: 1
bash (11216): drop_caches: 1
bash (13415): drop_caches: 1
bash (16791): drop_caches: 1
bash (18862): drop_caches: 1
bash (21442): drop_caches: 1
bash (23588): drop_caches: 1
bash (26576): drop_caches: 1
bash (28729): drop_caches: 1
bash (31230): drop_caches: 1
bash (1101): drop_caches: 1
bash (3819): drop_caches: 1
bash (6092): drop_caches: 1
bash (8770): drop_caches: 1
bash (10986): drop_caches: 1
bash (14022): drop_caches: 1
bash (16536): drop_caches: 1
bash (19119): drop_caches: 1
bash (21283): drop_caches: 1
bash (24060): drop_caches: 1
bash (26404): drop_caches: 1
bash (29576): drop_caches: 1
bash (31742): drop_caches: 1
bash (2158): drop_caches: 1
bash (4188): drop_caches: 1
bash (7360): drop_caches: 1
bash (9597): drop_caches: 1
bash (12683): drop_caches: 1
bash (15274): drop_caches: 1
bash (17451): drop_caches: 1
Lustre: 13233:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (20477): drop_caches: 1
bash (22667): drop_caches: 1
bash (25401): drop_caches: 1
bash (27567): drop_caches: 1
bash (30521): drop_caches: 1
bash (32540): drop_caches: 1
bash (3139): drop_caches: 1
bash (5432): drop_caches: 1
bash (8062): drop_caches: 1
bash (10078): drop_caches: 1
bash (13398): drop_caches: 1
bash (15919): drop_caches: 1
bash (18774): drop_caches: 1
bash (20757): drop_caches: 1
bash (23751): drop_caches: 1
bash (25843): drop_caches: 1
bash (28657): drop_caches: 1
bash (30789): drop_caches: 1
bash (1357): drop_caches: 1
bash (3385): drop_caches: 1
bash (5941): drop_caches: 1
bash (8044): drop_caches: 1
bash (10738): drop_caches: 1
bash (12908): drop_caches: 1
bash (15967): drop_caches: 1
bash (17921): drop_caches: 1
bash (20421): drop_caches: 1
bash (22301): drop_caches: 1
bash (25214): drop_caches: 1
bash (27248): drop_caches: 1
bash (30161): drop_caches: 1
bash (32303): drop_caches: 1
bash (2871): drop_caches: 1
bash (4860): drop_caches: 1
bash (7540): drop_caches: 1
bash (9364): drop_caches: 1
bash (12295): drop_caches: 1
bash (14642): drop_caches: 1
bash (17783): drop_caches: 1
bash (19833): drop_caches: 1
bash (21761): drop_caches: 1
bash (24367): drop_caches: 1
bash (26767): drop_caches: 1
bash (29367): drop_caches: 1
bash (31603): drop_caches: 1
bash (1832): drop_caches: 1
bash (3825): drop_caches: 1
bash (6551): drop_caches: 1
bash (8679): drop_caches: 1
bash (11557): drop_caches: 1
bash (13883): drop_caches: 1
bash (16858): drop_caches: 1
bash (18818): drop_caches: 1
bash (21482): drop_caches: 1
bash (23619): drop_caches: 1
bash (26675): drop_caches: 1
bash (28965): drop_caches: 1
bash (31603): drop_caches: 1
bash (1176): drop_caches: 1
bash (3954): drop_caches: 1
bash (6077): drop_caches: 1
bash (9046): drop_caches: 1
bash (11128): drop_caches: 1
bash (14542): drop_caches: 1
Lustre: 13233:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (16592): drop_caches: 1
bash (19131): drop_caches: 1
bash (21115): drop_caches: 1
bash (23842): drop_caches: 1
bash (25970): drop_caches: 1
bash (28846): drop_caches: 1
bash (30934): drop_caches: 1
bash (1255): drop_caches: 1
bash (3158): drop_caches: 1
bash (5933): drop_caches: 1
bash (8067): drop_caches: 1
bash (10641): drop_caches: 1
bash (12871): drop_caches: 1
bash (16351): drop_caches: 1
bash (18409): drop_caches: 1
bash (21197): drop_caches: 1
bash (23545): drop_caches: 1
bash (26580): drop_caches: 1
bash (28541): drop_caches: 1
bash (30529): drop_caches: 1
bash (758): drop_caches: 1
bash (3025): drop_caches: 1
bash (5814): drop_caches: 1
bash (7846): drop_caches: 1
bash (10314): drop_caches: 1
bash (12424): drop_caches: 1
bash (15711): drop_caches: 1
bash (17921): drop_caches: 1
bash (20932): drop_caches: 1
bash (23315): drop_caches: 1
bash (26337): drop_caches: 1
bash (28469): drop_caches: 1
bash (31501): drop_caches: 1
bash (1142): drop_caches: 1
bash (4005): drop_caches: 1
bash (6023): drop_caches: 1
bash (8774): drop_caches: 1
bash (10795): drop_caches: 1
bash (13862): drop_caches: 1
bash (16274): drop_caches: 1
bash (19026): drop_caches: 1
bash (21019): drop_caches: 1
bash (23493): drop_caches: 1
bash (25444): drop_caches: 1
bash (28136): drop_caches: 1
bash (29952): drop_caches: 1
bash (32465): drop_caches: 1
bash (2113): drop_caches: 1
bash (4733): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 45.1M 531M 0 116 0 1.09M
lustre-mdt1 45.1M 531M 0 0 0 975
lustre-mdt1 45.1M 531M 0 0 0 994
lustre-mdt1 45.1M 531M 0 0 0 992
lustre-mdt1 44.4M 532M 0 283 0 2.03M
lustre-mdt1 44.4M 532M 0 0 0 992
lustre-mdt1 44.4M 532M 0 0 0 992
lustre-mdt1 44.4M 532M 0 0 0 990
lustre-mdt1 44.4M 532M 0 0 0 993
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b19d067 PUD 31e849067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw i2c_core virtio_blk libata floppy [last unloaded: libcfs]
CPU: 15 PID: 982 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031fccb760 ti: ffff88031b988000 task.ti: ffff88031b988000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b98be50 EFLAGS: 00010246
RAX: ffff88031b98bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031b98be50 R08: 0000000000000000 R09: 00000000000008bd
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f6c3211e740(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000327a9e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

hrtimer: interrupt took 25000482 ns
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 54.5M 521M 0 431 0 6.82M
lustre-mdt1 55.6M 520M 0 161 0 3.16M
lustre-mdt1 57.2M 519M 0 252 0 4.55M
lustre-mdt1 55.3M 521M 0 446 0 6.50M
lustre-mdt1 55.3M 521M 0 148 0 2.44M
lustre-mdt1 55.0M 521M 0 375 0 5.90M
lustre-mdt1 52.7M 523M 0 216 0 2.71M
lustre-mdt1 53.2M 523M 0 158 0 2.85M
lustre-mdt1 52.3M 524M 0 310 0 3.46M
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31cf2e067 PUD 31cf2f067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw i2c_core virtio_blk floppy libata [last unloaded: libcfs]
CPU: 5 PID: 984 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031aee9280 ti: ffff88031a1fc000 task.ti: ffff88031a1fc000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a1ffe50 EFLAGS: 00010246
RAX: ffff88031a1fffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031a1ffe50 R08: 0000000000000000 R09: 00000000000062dd
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fa94a51a740(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031ce8a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff880270af8008: disconnect after 24s idle
Lustre: Skipped 3 previous similar messages
bash (9162): drop_caches: 1
bash (12340): drop_caches: 1
bash (14478): drop_caches: 1
bash (17460): drop_caches: 1
bash (19549): drop_caches: 1
bash (22622): drop_caches: 1
bash (24680): drop_caches: 1
bash (27863): drop_caches: 1
bash (30107): drop_caches: 1
bash (1131): drop_caches: 1
bash (3367): drop_caches: 1
bash (6141): drop_caches: 1
bash (8104): drop_caches: 1
bash (10577): drop_caches: 1
bash (12562): drop_caches: 1
bash (15377): drop_caches: 1
bash (17555): drop_caches: 1
bash (20864): drop_caches: 1
bash (23138): drop_caches: 1
bash (26351): drop_caches: 1
bash (28721): drop_caches: 1
bash (31920): drop_caches: 1
bash (1958): drop_caches: 1
bash (4977): drop_caches: 1
bash (7058): drop_caches: 1
bash (9960): drop_caches: 1
bash (12086): drop_caches: 1
bash (14637): drop_caches: 1
bash (16700): drop_caches: 1
bash (19384): drop_caches: 1
bash (21323): drop_caches: 1
Lustre: 30973:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (23338): drop_caches: 1
bash (26468): drop_caches: 1
bash (28460): drop_caches: 1
bash (31696): drop_caches: 1
bash (1806): drop_caches: 1
bash (4743): drop_caches: 1
bash (6926): drop_caches: 1
bash (9574): drop_caches: 1
bash (11898): drop_caches: 1
bash (14643): drop_caches: 1
bash (16887): drop_caches: 1
bash (19901): drop_caches: 1
bash (21987): drop_caches: 1
bash (24869): drop_caches: 1
bash (26966): drop_caches: 1
bash (30038): drop_caches: 1
bash (32651): drop_caches: 1
bash (3407): drop_caches: 1
bash (5461): drop_caches: 1
bash (8561): drop_caches: 1
bash (10743): drop_caches: 1
bash (13688): drop_caches: 1
bash (15719): drop_caches: 1
bash (18645): drop_caches: 1
bash (20934): drop_caches: 1
bash (24111): drop_caches: 1
bash (26302): drop_caches: 1
bash (29617): drop_caches: 1
bash (32094): drop_caches: 1
bash (2988): drop_caches: 1
bash (5201): drop_caches: 1
bash (7960): drop_caches: 1
bash (10031): drop_caches: 1
bash (13178): drop_caches: 1
bash (15377): drop_caches: 1
bash (18423): drop_caches: 1
bash (20584): drop_caches: 1
bash (23660): drop_caches: 1
bash (25871): drop_caches: 1
bash (28812): drop_caches: 1
bash (31193): drop_caches: 1
bash (2339): drop_caches: 1
bash (4560): drop_caches: 1
bash (6739): drop_caches: 1
bash (9744): drop_caches: 1
bash (11869): drop_caches: 1
bash (14959): drop_caches: 1
bash (17050): drop_caches: 1
bash (20177): drop_caches: 1
bash (22434): drop_caches: 1
bash (25062): drop_caches: 1
bash (27362): drop_caches: 1
bash (30442): drop_caches: 1
bash (575): drop_caches: 1
bash (3924): drop_caches: 1
bash (5995): drop_caches: 1
bash (8915): drop_caches: 1
bash (11165): drop_caches: 1
bash (14754): drop_caches: 1
bash (16946): drop_caches: 1
bash (20115): drop_caches: 1
bash (22361): drop_caches: 1
bash (25600): drop_caches: 1
bash (27895): drop_caches: 1
Lustre: 30973:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (31082): drop_caches: 1
bash (1301): drop_caches: 1
bash (4365): drop_caches: 1
bash (6562): drop_caches: 1
bash (9657): drop_caches: 1
bash (11617): drop_caches: 1
bash (14937): drop_caches: 1
bash (17131): drop_caches: 1
bash (20283): drop_caches: 1
bash (22352): drop_caches: 1
bash (25354): drop_caches: 1
bash (27711): drop_caches: 1
bash (30955): drop_caches: 1
bash (881): drop_caches: 1
bash (3804): drop_caches: 1
bash (5941): drop_caches: 1
bash (8727): drop_caches: 1
bash (10858): drop_caches: 1
bash (13960): drop_caches: 1
bash (16160): drop_caches: 1
bash (19312): drop_caches: 1
bash (21524): drop_caches: 1
bash (23736): drop_caches: 1
bash (26615): drop_caches: 1
bash (28858): drop_caches: 1
bash (31820): drop_caches: 1
bash (1801): drop_caches: 1
bash (4726): drop_caches: 1
bash (6925): drop_caches: 1
bash (10233): drop_caches: 1
bash (12471): drop_caches: 1
bash (15555): drop_caches: 1
bash (17893): drop_caches: 1
bash (20645): drop_caches: 1
bash (22894): drop_caches: 1
bash (25973): drop_caches: 1
bash (28186): drop_caches: 1
bash (31718): drop_caches: 1
bash (1955): drop_caches: 1
bash (5181): drop_caches: 1
bash (7402): drop_caches: 1
bash (10547): drop_caches: 1
bash (12902): drop_caches: 1
bash (16034): drop_caches: 1
bash (18180): drop_caches: 1
bash (21251): drop_caches: 1
bash (23322): drop_caches: 1
bash (26341): drop_caches: 1
bash (28345): drop_caches: 1
bash (31822): drop_caches: 1
bash (1960): drop_caches: 1
bash (4624): drop_caches: 1
bash (6774): drop_caches: 1
bash (9884): drop_caches: 1
bash (12021): drop_caches: 1
bash (14908): drop_caches: 1
bash (17080): drop_caches: 1
bash (19761): drop_caches: 1
bash (22050): drop_caches: 1
bash (24940): drop_caches: 1
bash (27044): drop_caches: 1
bash (30361): drop_caches: 1
bash (32630): drop_caches: 1
bash (3426): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 47.7M 528M 0 384 0 4.55M
lustre-mdt1 48.0M 528M 0 258 0 4.97M
lustre-mdt1 50.5M 526M 0 270 0 3.10M
lustre-mdt1 49.3M 527M 0 0 0 983
lustre-mdt1 50.8M 525M 0 252 0 2.17M
lustre-mdt1 45.0M 531M 0 252 0 1.59M
lustre-mdt1 45.0M 531M 0 0 0 994
lustre-mdt1 45.0M 531M 0 0 0 998
lustre-mdt1 45.0M 531M 0 0 0 993
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31a68d067 PUD 31a68c067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs]
CPU: 4 PID: 982 Comm: bash Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800bb8eb760 ti: ffff8800bb0e0000 task.ti: ffff8800bb0e0000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bb0e3e50 EFLAGS: 00010246
RAX: ffff8800bb0e3fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800bb0e3e50 R08: 0000000000000000 R09: 00000000000020f7
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fd110ec9740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319b18000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

Lustre: lustre-OST0000-osc-ffff8802ea0fd3d8: disconnect after 24s idle
Lustre: Skipped 8 previous similar messages
bash (15599): drop_caches: 1
bash (18223): drop_caches: 1
bash (21506): drop_caches: 1
bash (23774): drop_caches: 1
bash (26929): drop_caches: 1
bash (29179): drop_caches: 1
bash (32002): drop_caches: 1
bash (1534): drop_caches: 1
bash (4526): drop_caches: 1
bash (6572): drop_caches: 1
bash (8700): drop_caches: 1
bash (11198): drop_caches: 1
bash (13369): drop_caches: 1
bash (16088): drop_caches: 1
bash (18578): drop_caches: 1
bash (21225): drop_caches: 1
bash (23201): drop_caches: 1
bash (26163): drop_caches: 1
bash (28233): drop_caches: 1
bash (31171): drop_caches: 1
bash (786): drop_caches: 1
bash (3728): drop_caches: 1
bash (5630): drop_caches: 1
bash (8755): drop_caches: 1
bash (10933): drop_caches: 1
bash (13655): drop_caches: 1
bash (15883): drop_caches: 1
bash (18929): drop_caches: 1
bash (20949): drop_caches: 1
bash (23694): drop_caches: 1
bash (25684): drop_caches: 1
Lustre: 14568:0:(llog_cat.c:968:llog_cat_process_or_fork()) lustre-OST0000-osc-MDT0000: catlog [0x1:0x2:0x0] crosses index zero
bash (28526): drop_caches: 1
bash (30496): drop_caches: 1
bash (882): drop_caches: 1
bash (2899): drop_caches: 1
bash (5667): drop_caches: 1
bash (7656): drop_caches: 1
bash (10487): drop_caches: 1
bash (12463): drop_caches: 1
bash (15583): drop_caches: 1
bash (17730): drop_caches: 1
bash (20495): drop_caches: 1
bash (22415): drop_caches: 1
bash (25241): drop_caches: 1
bash (27386): drop_caches: 1
bash (30126): drop_caches: 1
bash (32163): drop_caches: 1
bash (2444): drop_caches: 1
bash (4503): drop_caches: 1
bash (7188): drop_caches: 1
bash (9327): drop_caches: 1
bash (11566): drop_caches: 1
bash (14464): drop_caches: 1
bash (16871): drop_caches: 1
bash (19451): drop_caches: 1
bash (21415): drop_caches: 1
bash (24017): drop_caches: 1
bash (26030): drop_caches: 1
bash (28791): drop_caches: 1
bash (30991): drop_caches: 1
bash (1149): drop_caches: 1
bash (3318): drop_caches: 1
bash (6065): drop_caches: 1
bash (8189): drop_caches: 1
bash (10991): drop_caches: 1
bash (12839): drop_caches: 1
bash (16314): drop_caches: 1
bash (18624): drop_caches: 1
bash (21544): drop_caches: 1
bash (23854): drop_caches: 1
bash (26696): drop_caches: 1
bash (28649): drop_caches: 1
bash (31458): drop_caches: 1
bash (820): drop_caches: 1
bash (3520): drop_caches: 1
bash (5449): drop_caches: 1
bash (8234): drop_caches: 1
bash (10403): drop_caches: 1
bash (13328): drop_caches: 1
bash (15657): drop_caches: 1
bash (18695): drop_caches: 1
bash (20709): drop_caches: 1
bash (23424): drop_caches: 1
bash (25144): drop_caches: 1
bash (27499): drop_caches: 1
bash (29327): drop_caches: 1
bash (32109): drop_caches: 1
bash (1937): drop_caches: 1
bash (4902): drop_caches: 1
bash (6973): drop_caches: 1
bash (10154): drop_caches: 1
bash (12189): drop_caches: 1
bash (15385): drop_caches: 1
bash (17474): drop_caches: 1
bash (19376): drop_caches: 1
bash (22009): drop_caches: 1
bash (24124): drop_caches: 1
bash (27051): drop_caches: 1
bash (29124): drop_caches: 1
bash (31595): drop_caches: 1
bash (1163): drop_caches: 1
bash (4034): drop_caches: 1
bash (6021): drop_caches: 1
bash (8799): drop_caches: 1
bash (10678): drop_caches: 1
bash (13266): drop_caches: 1
bash (15473): drop_caches: 1
bash (18521): drop_caches: 1
bash (20518): drop_caches: 1
bash (23603): drop_caches: 1
bash (25682): drop_caches: 1
bash (28541): drop_caches: 1
bash (30741): drop_caches: 1
bash (1150): drop_caches: 1
bash (3156): drop_caches: 1
bash (5546): drop_caches: 1
bash (7402): drop_caches: 1
bash (10421): drop_caches: 1
bash (12381): drop_caches: 1
bash (15599): drop_caches: 1
bash (18137): drop_caches: 1
bash (21101): drop_caches: 1
bash (23409): drop_caches: 1
bash (26486): drop_caches: 1
bash (28655): drop_caches: 1
bash (31769): drop_caches: 1
bash (1330): drop_caches: 1
bash (4201): drop_caches: 1
bash (6277): drop_caches: 1
bash (9100): drop_caches: 1
bash (11111): drop_caches: 1
bash (13798): drop_caches: 1
bash (16214): drop_caches: 1
bash (19551): drop_caches: 1
bash (21499): drop_caches: 1
bash (24380): drop_caches: 1
bash (26643): drop_caches: 1
bash (28730): drop_caches: 1
bash (31672): drop_caches: 1
bash (1277): drop_caches: 1
bash (4305): drop_caches: 1
bash (6601): drop_caches: 1
bash (9552): drop_caches: 1
bash (11855): drop_caches: 1
bash (14705): drop_caches: 1
bash (17067): drop_caches: 1
bash (19910): drop_caches: 1
bash (22394): drop_caches: 1
bash (25251): drop_caches: 1
bash (27406): drop_caches: 1
bash (30477): drop_caches: 1
bash (328): drop_caches: 1
bash (3472): drop_caches: 1
bash (5687): drop_caches: 1
bash (8941): drop_caches: 1
bash (11211): drop_caches: 1
bash (14209): drop_caches: 1
bash (17236): drop_caches: 1
bash (20322): drop_caches: 1
bash (22562): drop_caches: 1
bash (25717): drop_caches: 1
zpool stats on hang
capacity operations bandwidth
lustre-mdt1 48.2M 528M 0 798 0 7.78M
lustre-mdt1 49.8M 526M 0 479 0 4.98M
lustre-mdt1 51.6M 524M 0 184 0 2.39M
lustre-mdt1 45.2M 531M 0 621 0 6.58M
lustre-mdt1 39.3M 537M 0 46 0 120K
lustre-mdt1 39.3M 537M 0 0 0 1000
lustre-mdt1 39.3M 537M 0 0 0 995
lustre-mdt1 39.3M 537M 0 0 0 998
lustre-mdt1 39.3M 537M 0 0 0 997
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e08f067 PUD 31f273067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: obdecho]
CPU: 14 PID: 977 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800bba89280 ti: ffff88031fc50000 task.ti: ffff88031fc50000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fc53e50 EFLAGS: 00010246
RAX: ffff88031fc53fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d93a98 RDI: 0000000000000063
RBP: ffff88031fc53e50 R08: 0000000000000000 R09: 0000000000000acc
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fbe286e1740(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031bb9c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.8M 536M 0 0 0 995
lustre-mdt1 39.8M 536M 0 61 0 409K
lustre-mdt1 39.8M 536M 0 4 0 4.77K
lustre-mdt1 39.8M 536M 0 0 0 989
lustre-mdt1 39.8M 536M 0 0 0 991
lustre-mdt1 39.8M 536M 0 0 0 988
lustre-mdt1 39.7M 536M 0 64 0 365K
lustre-mdt1 39.7M 536M 0 0 0 986
lustre-mdt1 39.7M 536M 0 0 0 996
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD bb0b3067 PUD baae5067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy i2c_core libata [last unloaded: obdecho]
CPU: 4 PID: 979 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ab7a4f0 ti: ffff88031b02c000 task.ti: ffff88031b02c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031b02fe50 EFLAGS: 00010246
RAX: ffff88031b02ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b13a98 RDI: 0000000000000063
RBP: ffff88031b02fe50 R08: 0000000000000000 R09: 0000000000000ad3
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fdb6f43c740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031bb5a000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.5M 536M 0 71 0 333K
lustre-mdt1 39.5M 536M 0 0 0 998
lustre-mdt1 39.5M 536M 0 0 0 990
lustre-mdt1 39.5M 536M 0 0 0 996
lustre-mdt1 39.5M 536M 0 0 0 995
lustre-mdt1 39.6M 536M 0 63 0 388K
lustre-mdt1 39.6M 536M 0 0 0 992
lustre-mdt1 39.6M 536M 0 0 0 999
lustre-mdt1 39.6M 536M 0 0 0 991
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 327ab6067 PUD 319a75067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks serio_raw ata_piix virtio_blk i2c_core floppy libata [last unloaded: obdecho]
CPU: 12 PID: 982 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880322391280 ti: ffff88031fd64000 task.ti: ffff88031fd64000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031fd67e50 EFLAGS: 00010246
RAX: ffff88031fd67fd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331d13a98 RDI: 0000000000000063
RBP: ffff88031fd67e50 R08: 0000000000000000 R09: 0000000000000ac7
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f4915aa8740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000003198c6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.9M 536M 0 0 0 985
lustre-mdt1 39.8M 536M 0 83 0 418K
lustre-mdt1 39.8M 536M 0 0 0 992
lustre-mdt1 39.8M 536M 0 0 0 978
lustre-mdt1 39.8M 536M 0 0 0 991
lustre-mdt1 39.8M 536M 0 0 0 990
lustre-mdt1 39.8M 536M 0 63 0 375K
lustre-mdt1 39.8M 536M 0 0 0 994
lustre-mdt1 39.8M 536M 0 0 0 993
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31ba60067 PUD 31ba61067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw i2c_core libata [last unloaded: obdecho]
CPU: 9 PID: 974 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8800baa55c40 ti: ffff8800bab50000 task.ti: ffff8800bab50000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bab53e50 EFLAGS: 00010246
RAX: ffff8800bab53fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800bab53e50 R08: 0000000000000000 R09: 0000000000000aca
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fe07f03d740(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031d094000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.9M 536M 0 0 0 982
lustre-mdt1 39.9M 536M 0 0 0 979
lustre-mdt1 39.9M 536M 0 0 0 982
lustre-mdt1 39.9M 536M 0 85 0 370K
lustre-mdt1 39.9M 536M 0 0 0 988
lustre-mdt1 39.9M 536M 0 0 0 996
lustre-mdt1 39.9M 536M 0 0 0 994
lustre-mdt1 39.9M 536M 0 0 0 994
lustre-mdt1 39.8M 536M 0 73 0 417K
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3199d6067 PUD 3199d7067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix serio_raw drm_panel_orientation_quirks virtio_blk libata i2c_core floppy [last unloaded: obdecho]
CPU: 7 PID: 983 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e4fc9d0 ti: ffff880327a88000 task.ti: ffff880327a88000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880327a8be50 EFLAGS: 00010246
RAX: ffff880327a8bfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331bd3a98 RDI: 0000000000000063
RBP: ffff880327a8be50 R08: 0000000000000000 R09: 0000000000000bd4
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f1d18ba4740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031bb3e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.4M 537M 0 78 0 365K
lustre-mdt1 39.4M 537M 0 0 0 986
lustre-mdt1 39.4M 537M 0 0 0 982
lustre-mdt1 39.4M 537M 0 0 0 986
lustre-mdt1 39.4M 537M 0 0 0 984
lustre-mdt1 39.4M 537M 0 76 0 394K
lustre-mdt1 39.4M 537M 0 0 0 987
lustre-mdt1 39.4M 537M 0 0 0 973
lustre-mdt1 39.4M 537M 0 0 0 983
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31f192067 PUD 31cef3067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata [last unloaded: obdecho]
CPU: 4 PID: 959 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031df79280 ti: ffff88031ae70000 task.ti: ffff88031ae70000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031ae73e50 EFLAGS: 00010246
RAX: ffff88031ae73fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031ae73e50 R08: 0000000000000000 R09: 0000000000000ad4
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f904adea740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031d1b2000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.8M 536M 0 0 0 998
lustre-mdt1 39.8M 536M 0 0 0 997
lustre-mdt1 39.9M 536M 0 84 0 512K
lustre-mdt1 39.9M 536M 0 0 0 994
lustre-mdt1 39.9M 536M 0 0 0 996
lustre-mdt1 39.9M 536M 0 0 0 999
lustre-mdt1 39.9M 536M 0 0 0 994
lustre-mdt1 39.9M 536M 0 69 0 413K
lustre-mdt1 39.9M 536M 0 0 0 995
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31ba8b067 PUD 31b00f067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw i2c_core virtio_blk libata floppy [last unloaded: obdecho]
CPU: 1 PID: 965 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff8803195c24f0 ti: ffff8800bb094000 task.ti: ffff8800bb094000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800bb097e50 EFLAGS: 00010246
RAX: ffff8800bb097fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800bb097e50 R08: 0000000000000000 R09: 0000000000000ae2
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f34b4b8b740(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000003220fa000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.9M 536M 0 71 0 370K
lustre-mdt1 39.9M 536M 0 0 0 999
lustre-mdt1 39.9M 536M 0 0 0 992
lustre-mdt1 39.9M 536M 0 0 0 992
lustre-mdt1 39.9M 536M 0 0 0 995
lustre-mdt1 39.9M 536M 0 64 0 426K
lustre-mdt1 39.9M 536M 0 1 0 1.95K
lustre-mdt1 39.9M 536M 0 0 0 999
lustre-mdt1 39.9M 536M 0 0 0 998
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 327a9b067 PUD 31a854067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk i2c_core libata serio_raw floppy [last unloaded: obdecho]
CPU: 0 PID: 993 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031e81a4f0 ti: ffff88031bed8000 task.ti: ffff88031bed8000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031bedbe50 EFLAGS: 00010246
RAX: ffff88031bedbfd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331a13a98 RDI: 0000000000000063
RBP: ffff88031bedbe50 R08: 0000000000000000 R09: 0000000000000bfe
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fd7416ed740(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 00000003198e4000 CR4: 00000000000007f0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 21.0M 555M 0 0 0 978
lustre-mdt1 21.0M 555M 0 0 0 986
lustre-mdt1 21.0M 555M 0 0 0 987
lustre-mdt1 21.0M 555M 0 0 0 971
lustre-mdt1 20.9M 555M 0 53 0 398K
lustre-mdt1 20.9M 555M 0 0 0 991
lustre-mdt1 20.9M 555M 0 0 0 977
lustre-mdt1 20.9M 555M 0 0 0 978
lustre-mdt1 20.9M 555M 0 0 0 986
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e64d067 PUD 31e64c067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix libata serio_raw i2c_core floppy virtio_blk [last unloaded: obdecho]
CPU: 7 PID: 994 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031b950010 ti: ffff88031af6c000 task.ti: ffff88031af6c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031af6fe50 EFLAGS: 00010246
RAX: ffff88031af6ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331bd3a98 RDI: 0000000000000063
RBP: ffff88031af6fe50 R08: 0000000000000000 R09: 0000000000000ad7
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fcee8b19740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031adce000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.8M 536M 0 0 0 994
lustre-mdt1 39.9M 536M 0 58 0 430K
lustre-mdt1 39.9M 536M 0 1 0 1.93K
lustre-mdt1 39.9M 536M 0 0 0 973
lustre-mdt1 39.9M 536M 0 0 0 980
lustre-mdt1 39.9M 536M 0 0 0 948
lustre-mdt1 39.9M 536M 0 72 0 406K
lustre-mdt1 39.9M 536M 0 0 0 987
lustre-mdt1 39.9M 536M 0 0 0 993
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31e89c067 PUD 31b234067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm serio_raw drm_panel_orientation_quirks ata_piix floppy virtio_blk libata i2c_core [last unloaded: obdecho]
CPU: 4 PID: 1008 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031ab63760 ti: ffff88031a41c000 task.ti: ffff88031a41c000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031a41fe50 EFLAGS: 00010246
RAX: ffff88031a41ffd8 RBX: ffffffff81d17f60 RCX: 0000000000000000
RDX: 0000000000000000 RSI: ffff880331b13a98 RDI: 0000000000000063
RBP: ffff88031a41fe50 R08: 0000000000000000 R09: 0000000000000ad9
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f8b0421b740(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319ab6000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.6M 536M 0 0 0 965
lustre-mdt1 39.6M 536M 0 59 0 322K
lustre-mdt1 39.6M 536M 0 0 0 977
lustre-mdt1 39.6M 536M 0 0 0 953
lustre-mdt1 39.6M 536M 0 0 0 980
lustre-mdt1 39.6M 536M 0 0 0 990
lustre-mdt1 39.6M 536M 0 72 0 398K
lustre-mdt1 39.6M 536M 0 0 0 974
lustre-mdt1 39.6M 536M 0 0 0 952
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3220f1067 PUD 31e09c067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk i2c_core serio_raw floppy [last unloaded: obdecho]
CPU: 12 PID: 959 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031d518010 ti: ffff880327a50000 task.ti: ffff880327a50000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff880327a53e50 EFLAGS: 00010246
RAX: ffff880327a53fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff880327a53e50 R08: 0000000000000000 R09: 0000000000000ae2
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007fd3de55b740(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031df1c000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.7M 536M 0 0 0 991
lustre-mdt1 39.7M 536M 0 0 0 970
lustre-mdt1 39.7M 536M 0 1 0 1.93K
lustre-mdt1 39.7M 536M 0 63 0 394K
lustre-mdt1 39.7M 536M 0 0 0 995
lustre-mdt1 39.7M 536M 0 0 0 970
lustre-mdt1 39.7M 536M 0 0 0 994
lustre-mdt1 39.7M 536M 0 0 0 974
lustre-mdt1 39.6M 536M 0 57 0 325K
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 31b05e067 PUD 31b05f067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core floppy libata [last unloaded: obdecho]
CPU: 7 PID: 1004 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff880319df8010 ti: ffff88031e5a4000 task.ti: ffff88031e5a4000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff88031e5a7e50 EFLAGS: 00010246
RAX: ffff88031e5a7fd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff88031e5a7e50 R08: 0000000000000000 R09: 0000000000000ad0
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f0b22b84740(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000319d7e000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.6M 536M 0 0 0 988
lustre-mdt1 39.7M 536M 0 77 0 407K
lustre-mdt1 39.7M 536M 0 0 0 994
lustre-mdt1 39.7M 536M 0 0 0 979
lustre-mdt1 39.7M 536M 0 0 0 986
lustre-mdt1 39.7M 536M 0 11 0 93.9K
lustre-mdt1 39.7M 536M 0 71 0 370K
lustre-mdt1 39.7M 536M 0 0 0 992
lustre-mdt1 39.7M 536M 0 0 0 973
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash

BUG: unable to handle kernel NULL pointer dereference at (null)
IP: [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
PGD 3196e5067 PUD 320138067 PMD 0
Oops: 0002 [#1] SMP DEBUG_PAGEALLOC
Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix virtio_blk i2c_core serio_raw floppy libata [last unloaded: obdecho]
CPU: 3 PID: 975 Comm: bash Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2
Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014
task: ffff88031d1e49d0 ti: ffff8800baecc000 task.ti: ffff8800baecc000
RIP: 0010:[<ffffffff814d87e6>] [<ffffffff814d87e6>] sysrq_handle_crash+0x26/0x30
RSP: 0018:ffff8800baecfe50 EFLAGS: 00010246
RAX: ffff8800baecffd8 RBX: ffffffff81d17f60 RCX: 0000000000000006
RDX: 0000000000000000 RSI: 0000000000000246 RDI: 0000000000000063
RBP: ffff8800baecfe50 R08: 0000000000000000 R09: 0000000000000ae5
R10: 000000000000ffa0 R11: 0000000000000198 R12: 0000000000000063
R13: 0000000000000000 R14: 0000000000000007 R15: 0000000000000000
FS: 00007f020cfbb740(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000031b344000 CR4: 00000000000007e0
Call Trace:
[<ffffffff814d9027>] __handle_sysrq+0x117/0x180
[<ffffffff814d94ba>] write_sysrq_trigger+0x4a/0x60
[<ffffffff812c29ad>] proc_reg_write+0x3d/0x80
[<ffffffff81245a16>] vfs_write+0xd6/0x230
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff8124685f>] SyS_write+0x7f/0xf0
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a
[<ffffffff817ee00c>] system_call_fastpath+0x1f/0x24
[<ffffffff817edf55>] ? system_call_after_swapgs+0xa2/0x13a

zpool stats on hang
capacity operations bandwidth
lustre-mdt1 39.5M 536M 0 0 0 989
lustre-mdt1 39.5M 536M 0 0 0 981
lustre-mdt1 39.5M 536M 0 75 0 437K
lustre-mdt1 39.5M 536M 0 0 0 999
lustre-mdt1 39.5M 536M 0 0 0 997
lustre-mdt1 39.5M 536M 0 0 0 981
lustre-mdt1 39.5M 536M 0 0 0 992
lustre-mdt1 39.7M 536M 0 72 0 514K
lustre-mdt1 39.7M 536M 0 0 0 994
Lustre: OBD: obd_test_cleanup
SysRq :
Trigger a crash