Edit new crash report

Reason	Crashing Function	Where to cut Backtrace	Reports Count
BUG: unable to handle kernel NULL pointer dereference	qsd_op_end	osd_trans_stop mgs_nidtbl_write mgs_ir_update mgs_target_reg tgt_request_handle ptlrpc_server_handle_request ptlrpc_main kthread ret_from_fork	7

Failing Test	Full Crash	Messages before crash	Comment
recovery-small test 107: drop reint reply, then restart MDT	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 1 PID: 163776 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.137.1.el8_10_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 4f cc 05 00 01 00 00 00 48 c7 05 4c cc 05 00 00 00 00 00 e8 27 06 2a ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 34 2b ff 01 0f 84 7b 01 00 RSP: 0018:ff6737b0c22a7ad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff4049b8c2d88800 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff4049b8c3d21e80 RBP: ffffffffc1488300 R08: 0000000000000f94 R09: 0000000000000000 R10: ff4049b8f4bb5000 R11: ff4049b8f4bb4f00 R12: ff4049b8edd18ac0 R13: ff4049b8e2349b88 R14: ffffffffc148b310 R15: ff4049b8c1475f88 FS: 0000000000000000(0000) GS:ff4049b93bb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 0000000142e10005 CR4: 0000000000771ee0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_add+0x117/0x180 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common dns_resolver nfs lockd grace fscache kvm_amd iTCO_wdt iTCO_vendor_support ccp kvm joydev irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr lpc_ich i2c_i801 virtio_balloon sunrpc ext4 mbcache jbd2 crc32c_intel ahci libahci libata serio_raw virtio_net virtio_blk net_failover failover CR2: 0000000000000060	Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 Lustre: * cfs_fail_loc=119, val=2147483648* LustreError: 9785:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ff4049b8ec369040 x1871254464566528/t94489280516(0) o36->5665f5f1-6c58-44fb-b986-83777f50f6be@10.240.47.19@tcp:697/0 lens 616/448 e 0 to 0 dl 1784571547 ref 1 fl Interpret:/200/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 Lustre: DEBUG MARKER: lctl set_param fail_loc=0 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 10.240.47.21@tcp (stopping) Lustre: Skipped 1 previous similar message	Link to test
recovery-small test 111: mdd setup fail should not cause umount oops	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 1 PID: 2810927 Comm: ll_mgs_0002 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 27 d6 36 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 04 38 ff 01 0f 84 7b 01 00 RSP: 0018:ff2e9115c7ecbad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff2e11d922003200 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff2e11d9335fa080 RBP: ffffffffc14cb300 R08: 0000000000000298 R09: 0000000000000000 R10: ff2e11d8ecbac000 R11: ff2e11d8ecbab200 R12: ff2e11d8f035dc40 R13: ff2e11d8f0861b88 R14: ffffffffc14ce318 R15: ff2e11d8c6eb0ab8 FS: 0000000000000000(0000) GS:ff2e11d93bb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 000000000fc10001 CR4: 0000000000771ee0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_add+0x117/0x180 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common nfsv4 kvm_amd dns_resolver nfs lockd grace fscache ccp kvm iTCO_wdt iTCO_vendor_support joydev i2c_i801 irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr lpc_ich virtio_balloon sunrpc ext4 mbcache jbd2 ahci libahci libata crc32c_intel virtio_net net_failover failover serio_raw virtio_blk [last unloaded: libcfs] CR2: 0000000000000060	Lustre: DEBUG MARKER: lctl set_param fail_loc=0x151 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LustreError: 2333413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 10.240.39.212@tcp arrived at 1783410497 with bad export cookie 11607815987956998942 Lustre: lustre-MDT0000: Not available for connect from 10.240.39.212@tcp (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 2305595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 2305595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 74 previous similar messages Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC10.240.39.211@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: * cfs_fail_loc=151, val=0* LustreError: 2810919:0:(mdd_device.c:673:mdd_changelog_init()) lustre-MDD0000: changelog setup during init failed: rc = -5 LustreError: 2810919:0:(mdd_device.c:1406:mdd_prepare()) lustre-MDD0000: failed to initialize changelog: rc = -5 LustreError: 2810919:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -5 Lustre: Failing over lustre-MDT0000 LustreError: 2810972:0:(llog_osd.c:225:llog_osd_read_header()) lustre-MDT0001-osp-MDT0000: can't read llog [0x240000bdc:0x1:0x0] header: rc = -5 Lustre: 2810972:0:(llog_cat.c:847:llog_cat_process_common()) lustre-MDT0001-osp-MDT0000: can't find llog handle [0x240000bdc:0x1:0x0]: rc = -5 LustreError: 2810972:0:(llog.c:870:llog_process_thread()) lustre-MDT0001-osp-MDT0000 retry remote llog process LustreError: 2810972:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -11	Link to test
recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 1 PID: 39832 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 cf de 05 00 01 00 00 00 48 c7 05 cc de 05 00 00 00 00 00 e8 37 b8 66 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e e4 67 ff 01 0f 84 7b 01 00 RSP: 0018:ff551d9e04b67ad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff45e4a488e5c600 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff45e4a4834bcf80 RBP: ffffffffc1393300 R08: 00000000000007e4 R09: 0000000000000000 R10: ff45e4a4adba1000 R11: ff45e4a4adba0789 R12: ff45e4a4892c3980 R13: ff45e4a482a51b88 R14: ffffffffc1396330 R15: ff45e4a48161ad90 FS: 0000000000000000(0000) GS:ff45e4a4fbb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 0000000170010002 CR4: 0000000000771ee0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_add+0x117/0x180 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mgs(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common kvm_amd nfsv4 ccp dns_resolver kvm nfs lockd grace fscache irqbypass crct10dif_pclmul iTCO_wdt iTCO_vendor_support crc32_pclmul ghash_clmulni_intel pcspkr joydev virtio_balloon lpc_ich i2c_i801 sunrpc dm_mod ext4 mbcache jbd2 ahci libahci libata crc32c_intel serio_raw virtio_net net_failover virtio_blk failover CR2: 0000000000000060	Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LustreError: 41646:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery Lustre: 39876:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: 39876:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages LustreError: 39876:0:(ldlm_lib.c:1920:abort_lock_replay_queue()) @@@ aborted: req@ff45e4a4b87a4340 x1869733319495040/t0(0) o101->lustre-MDT0003-mdtlov_UUID@10.240.28.103@tcp:742/0 lens 328/0 e 0 to 0 dl 1783121992 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000-osd: cancel update llog [0x200009870:0x1:0x0] Lustre: lustre-MDT0000: Not available for connect from 10.240.28.103@tcp (stopping) Lustre: Skipped 28 previous similar messages Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000409:0x1:0x0] LustreError: 39876:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff45e4a4b82dd380 x1869738608583424/t0(0) o700->lustre-MDT0001-osp-MDT0000@10.240.28.103@tcp:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'tgt_recover_0.0' uid:0 gid:0 projid:4294967295 LustreError: 39876:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 39876:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x280000407:0x1:0x0] Lustre: lustre-MDT0003-osp-MDT0000: cancel update llog [0x2c0000408:0x1:0x0] Lustre: lustre-MDT0000: Recovery over after 0:03, of 5 clients 0 recovered and 5 were evicted. LustreError: 41647:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x20000e692:0x2:0x0]: rc = -2	Link to test
recovery-small test 111: mdd setup fail should not cause umount oops	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 177918 Comm: ll_mgs_0002 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 37 f8 11 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 24 13 ff 01 0f 84 7b 01 00 RSP: 0018:ff68afa207c73ad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff2adcb76ef1ac00 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff2adcb745901300 RBP: ffffffffc132b300 R08: 0000000000000e27 R09: 0000000000000000 R10: ff2adcb76f3a3000 R11: ff2adcb76f3a2dcc R12: ff2adcb77684d0c0 R13: ff2adcb76b382b88 R14: ffffffffc132e310 R15: ff2adcb7466b5b28 FS: 0000000000000000(0000) GS:ff2adcb7bba00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 0000000134210006 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_add+0x117/0x180 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod intel_rapl_msr intel_rapl_common rpcsec_gss_krb5 auth_rpcgss kvm_amd ccp nfsv4 iTCO_wdt dns_resolver iTCO_vendor_support kvm nfs lockd grace fscache irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev i2c_i801 pcspkr lpc_ich virtio_balloon sunrpc ext4 mbcache jbd2 ahci libahci virtio_net crc32c_intel libata serio_raw net_failover failover virtio_blk CR2: 0000000000000060	Lustre: DEBUG MARKER: lctl set_param fail_loc=0x151 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC10.240.23.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: * cfs_fail_loc=151, val=0* LustreError: 177910:0:(mdd_device.c:673:mdd_changelog_init()) lustre-MDD0000: changelog setup during init failed: rc = -5 LustreError: 177910:0:(mdd_device.c:1406:mdd_prepare()) lustre-MDD0000: failed to initialize changelog: rc = -5 LustreError: 177910:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -5 Lustre: Failing over lustre-MDT0000 LustreError: 177958:0:(llog_osd.c:225:llog_osd_read_header()) lustre-MDT0001-osp-MDT0000: can't read llog [0x24000040b:0x1:0x0] header: rc = -5 Lustre: 177958:0:(llog_cat.c:847:llog_cat_process_common()) lustre-MDT0001-osp-MDT0000: can't find llog handle [0x24000040b:0x1:0x0]: rc = -5 LustreError: 177958:0:(llog.c:870:llog_process_thread()) lustre-MDT0001-osp-MDT0000 retry remote llog process LustreError: 177958:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -11 Lustre: lustre-MDT0000: Not available for connect from 10.240.23.111@tcp (stopping) Lustre: Skipped 7 previous similar messages	Link to test
conf-sanity test 5i: start mdt failure at mdt_quota_init()	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 1 PID: 73233 Comm: ll_mgs_0002 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 37 18 13 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 44 14 ff 01 0f 84 7b 01 00 RSP: 0018:ff876f1941b4fad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff4d49442acf3000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff4d49440436eb80 RBP: ffffffffc143b300 R08: 0000000000000e50 R09: 0000000000000000 R10: ff4d494435c38000 R11: ff4d494435c37d00 R12: ff4d49442a29ec00 R13: ff4d4944276c3b88 R14: ffffffffc143e310 R15: ff4d494401476b28 FS: 0000000000000000(0000) GS:ff4d49447bb00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 0000000042a10005 CR4: 0000000000771ee0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_add+0x117/0x180 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod intel_rapl_msr intel_rapl_common kvm_amd ccp rpcsec_gss_krb5 auth_rpcgss kvm nfsv4 dns_resolver nfs lockd grace fscache iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i2c_i801 joydev pcspkr lpc_ich virtio_balloon sunrpc ext4 mbcache jbd2 ahci libahci virtio_net libata crc32c_intel serio_raw net_failover failover virtio_blk CR2: 0000000000000060	Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm118.onyx.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm118.onyx.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: onyx-155vm118.onyx.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: onyx-155vm118.onyx.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm116.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm115.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-155vm116.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-155vm115.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm117.onyx.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: onyx-155vm117.onyx.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n version 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm116.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-155vm115.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: onyx-155vm116.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: onyx-155vm115.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000A05 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: * cfs_fail_loc=a05, val=0* Lustre: lustre-MDT0000: Not available for connect from 10.240.31.121@tcp (not set up) Lustre: Skipped 1 previous similar message LustreError: 73256:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-9) LustreError: 73256:0:(obd_config.c:2103:class_config_llog_handler()) MGC10.240.31.122@tcp: cfg command failed: rc = -9	Link to test
recovery-small test 107: drop reint reply, then restart MDT	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 163781 Comm: ll_mgs_0000 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 27 b8 11 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e e4 12 ff 01 0f 84 7b 01 00 RSP: 0018:ff611844023ffad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff11e7163871ba00 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff11e716033d2a00 RBP: ffffffffc1229300 R08: 0000000000000342 R09: 0000000000000000 R10: ff11e71637ae2000 R11: ff11e71637ae12e7 R12: ff11e7162cecea40 R13: ff11e7163558cb88 R14: ffffffffc122c310 R15: ff11e7160150dc08 FS: 0000000000000000(0000) GS:ff11e7167ba00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 0000000020210003 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? lu_env_refill+0xc/0x50 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? srso_alias_return_thunk+0x5/0xfcdfd ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common dns_resolver kvm_amd ccp nfs lockd grace fscache kvm iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr i2c_i801 virtio_balloon lpc_ich sunrpc ext4 mbcache jbd2 ahci libahci libata crc32c_intel serio_raw virtio_net net_failover virtio_blk failover CR2: 0000000000000060	Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119 Lustre: * cfs_fail_loc=119, val=2147483648* LustreError: 37585:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ff11e7163750aa40 x1868103326105856/t94489280516(0) o36->a2c32b26-4eda-461a-89ad-c0dfdfab101a@10.240.39.217@tcp:565/0 lens 616/448 e 0 to 0 dl 1781566515 ref 1 fl Interpret:/200/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295 Lustre: DEBUG MARKER: lctl set_param fail_loc=0 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 10.240.39.219@tcp (stopping) Lustre: Skipped 1 previous similar message	Link to test
conf-sanity test 5h: start mdt failure at mdt_fs_setup()	BUG: unable to handle kernel NULL pointer dereference at 0000000000000060 PGD 0 Oops: 0000 [#1] SMP NOPTI CPU: 0 PID: 145989 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 RIP: 0010:qsd_op_end+0x99/0x340 [lquota] Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 27 38 51 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 64 52 ff 01 0f 84 7b 01 00 RSP: 0018:ff53629701977ad0 EFLAGS: 00010286 RAX: 0000000000000000 RBX: ff3c09aabd335e00 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff3c09aa82c7e200 RBP: ffffffffc15e6300 R08: 0000000000000ec5 R09: 0000000000000000 R10: ff3c09aabf164000 R11: ff3c09aabf163e6a R12: ff3c09aa8dafd440 R13: ff3c09aaaec61b88 R14: ffffffffc15e9310 R15: ff3c09aa816284d0 FS: 0000000000000000(0000) GS:ff3c09aafba00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000060 CR3: 0000000139e10006 CR4: 0000000000771ef0 PKRU: 55555554 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? qsd_op_end+0x99/0x340 [lquota] osd_trans_stop+0x363/0xaa0 [osd_ldiskfs] ? srso_alias_return_thunk+0x5/0xfcdfd ? osd_write+0x148/0x700 [osd_ldiskfs] mgs_nidtbl_write+0x949/0xb20 [mgs] mgs_ir_update+0x79/0x2a0 [mgs] mgs_target_reg+0xcd3/0x1eb0 [mgs] ? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc] ? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc] tgt_request_handle+0x403/0x1d00 [ptlrpc] ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc] ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_add+0x117/0x180 [obdclass] ptlrpc_main+0xb90/0x1450 [ptlrpc] ? __schedule+0x2d9/0x870 ? srso_alias_return_thunk+0x5/0xfcdfd ? ptlrpc_wait_event+0x390/0x390 [ptlrpc] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod intel_rapl_msr intel_rapl_common kvm_amd rpcsec_gss_krb5 auth_rpcgss ccp kvm nfsv4 dns_resolver nfs lockd grace fscache iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel pcspkr virtio_balloon i2c_i801 lpc_ich sunrpc ext4 mbcache jbd2 ahci libahci virtio_net crc32c_intel serio_raw libata net_failover virtio_blk failover CR2: 0000000000000060	Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null \| grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm182.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-156vm182.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\.ost_server_uuid Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) osc.lustre-OST0000-osc-[-0-9a-f].ost_server_uuid Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts \|\| true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000135 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 9 previous similar messages Lustre: * cfs_fail_loc=135, val=0* LustreError: 145993:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2) LustreError: 145993:0:(obd_config.c:2103:class_config_llog_handler()) MGC10.240.24.31@tcp: cfg command failed: rc = -2	Link to test

Failing Test

Full Crash

Messages before crash

Comment

recovery-small test 107: drop reint reply, then restart MDT

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 1 PID: 163776 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.137.1.el8_10_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 4f cc 05 00 01 00 00 00 48 c7 05 4c cc 05 00 00 00 00 00 e8 27 06 2a ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 34 2b ff 01 0f 84 7b 01 00
RSP: 0018:ff6737b0c22a7ad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff4049b8c2d88800 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff4049b8c3d21e80
RBP: ffffffffc1488300 R08: 0000000000000f94 R09: 0000000000000000
R10: ff4049b8f4bb5000 R11: ff4049b8f4bb4f00 R12: ff4049b8edd18ac0
R13: ff4049b8e2349b88 R14: ffffffffc148b310 R15: ff4049b8c1475f88
FS: 0000000000000000(0000) GS:ff4049b93bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000000142e10005 CR4: 0000000000771ee0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common dns_resolver nfs lockd grace fscache kvm_amd iTCO_wdt iTCO_vendor_support ccp kvm joydev irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr lpc_ich i2c_i801 virtio_balloon sunrpc ext4 mbcache jbd2 crc32c_intel ahci libahci libata serio_raw virtio_net virtio_blk net_failover failover
CR2: 0000000000000060

Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 9785:0:(ldlm_lib.c:3345:target_send_reply_msg()) @@@ dropping reply req@ff4049b8ec369040 x1871254464566528/t94489280516(0) o36->5665f5f1-6c58-44fb-b986-83777f50f6be@10.240.47.19@tcp:697/0 lens 616/448 e 0 to 0 dl 1784571547 ref 1 fl Interpret:/200/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295
Lustre: DEBUG MARKER: lctl set_param fail_loc=0
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
Lustre: lustre-MDT0000: Not available for connect from 10.240.47.21@tcp (stopping)
Lustre: Skipped 1 previous similar message

Link to test

recovery-small test 111: mdd setup fail should not cause umount oops

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 1 PID: 2810927 Comm: ll_mgs_0002 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 27 d6 36 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 04 38 ff 01 0f 84 7b 01 00
RSP: 0018:ff2e9115c7ecbad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff2e11d922003200 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff2e11d9335fa080
RBP: ffffffffc14cb300 R08: 0000000000000298 R09: 0000000000000000
R10: ff2e11d8ecbac000 R11: ff2e11d8ecbab200 R12: ff2e11d8f035dc40
R13: ff2e11d8f0861b88 R14: ffffffffc14ce318 R15: ff2e11d8c6eb0ab8
FS: 0000000000000000(0000) GS:ff2e11d93bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 000000000fc10001 CR4: 0000000000771ee0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common nfsv4 kvm_amd dns_resolver nfs lockd grace fscache ccp kvm iTCO_wdt iTCO_vendor_support joydev i2c_i801 irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel pcspkr lpc_ich virtio_balloon sunrpc ext4 mbcache jbd2 ahci libahci libata crc32c_intel virtio_net net_failover failover serio_raw virtio_blk [last unloaded: libcfs]
CR2: 0000000000000060

Lustre: DEBUG MARKER: lctl set_param fail_loc=0x151
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 2333413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 10.240.39.212@tcp arrived at 1783410497 with bad export cookie 11607815987956998942
Lustre: lustre-MDT0000: Not available for connect from 10.240.39.212@tcp (stopping)
Lustre: Skipped 1 previous similar message
Lustre: server umount lustre-MDT0000 complete
LustreError: 2305595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server.
LustreError: 2305595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 74 previous similar messages
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
LustreError: MGC10.240.39.211@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 2 previous similar messages
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 2 previous similar messages
Lustre: *** cfs_fail_loc=151, val=0***
LustreError: 2810919:0:(mdd_device.c:673:mdd_changelog_init()) lustre-MDD0000: changelog setup during init failed: rc = -5
LustreError: 2810919:0:(mdd_device.c:1406:mdd_prepare()) lustre-MDD0000: failed to initialize changelog: rc = -5
LustreError: 2810919:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -5
Lustre: Failing over lustre-MDT0000
LustreError: 2810972:0:(llog_osd.c:225:llog_osd_read_header()) lustre-MDT0001-osp-MDT0000: can't read llog [0x240000bdc:0x1:0x0] header: rc = -5
Lustre: 2810972:0:(llog_cat.c:847:llog_cat_process_common()) lustre-MDT0001-osp-MDT0000: can't find llog handle [0x240000bdc:0x1:0x0]: rc = -5
LustreError: 2810972:0:(llog.c:870:llog_process_thread()) lustre-MDT0001-osp-MDT0000 retry remote llog process
LustreError: 2810972:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -11

Link to test

recovery-small test 143: orphan cleanup thread shouldn't be blocked even delete failed

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 1 PID: 39832 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 cf de 05 00 01 00 00 00 48 c7 05 cc de 05 00 00 00 00 00 e8 37 b8 66 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e e4 67 ff 01 0f 84 7b 01 00
RSP: 0018:ff551d9e04b67ad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff45e4a488e5c600 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff45e4a4834bcf80
RBP: ffffffffc1393300 R08: 00000000000007e4 R09: 0000000000000000
R10: ff45e4a4adba1000 R11: ff45e4a4adba0789 R12: ff45e4a4892c3980
R13: ff45e4a482a51b88 R14: ffffffffc1396330 R15: ff45e4a48161ad90
FS: 0000000000000000(0000) GS:ff45e4a4fbb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000000170010002 CR4: 0000000000771ee0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mgs(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr intel_rapl_common kvm_amd nfsv4 ccp dns_resolver kvm nfs lockd grace fscache irqbypass crct10dif_pclmul iTCO_wdt iTCO_vendor_support crc32_pclmul ghash_clmulni_intel pcspkr joydev virtio_balloon lpc_ich i2c_i801 sunrpc dm_mod ext4 mbcache jbd2 ahci libahci libata crc32c_intel serio_raw virtio_net net_failover virtio_blk failover
CR2: 0000000000000060

Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
LustreError: 41646:0:(ldlm_lib.c:2990:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery
Lustre: 39876:0:(ldlm_lib.c:2390:target_recovery_overseer()) recovery is aborted, evict exports in recovery
Lustre: 39876:0:(ldlm_lib.c:2390:target_recovery_overseer()) Skipped 2 previous similar messages
LustreError: 39876:0:(ldlm_lib.c:1920:abort_lock_replay_queue()) @@@ aborted: req@ff45e4a4b87a4340 x1869733319495040/t0(0) o101->lustre-MDT0003-mdtlov_UUID@10.240.28.103@tcp:742/0 lens 328/0 e 0 to 0 dl 1783121992 ref 1 fl Complete:/240/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' uid:0 gid:0 projid:4294967295
Lustre: lustre-MDT0000-osd: cancel update llog [0x200009870:0x1:0x0]
Lustre: lustre-MDT0000: Not available for connect from 10.240.28.103@tcp (stopping)
Lustre: Skipped 28 previous similar messages
Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000409:0x1:0x0]
LustreError: 39876:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ff45e4a4b82dd380 x1869738608583424/t0(0) o700->lustre-MDT0001-osp-MDT0000@10.240.28.103@tcp:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'tgt_recover_0.0' uid:0 gid:0 projid:4294967295
LustreError: 39876:0:(fid_request.c:217:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5
LustreError: 39876:0:(fid_request.c:321:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5
Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x280000407:0x1:0x0]
Lustre: lustre-MDT0003-osp-MDT0000: cancel update llog [0x2c0000408:0x1:0x0]
Lustre: lustre-MDT0000: Recovery over after 0:03, of 5 clients 0 recovered and 5 were evicted.
LustreError: 41647:0:(osd_handler.c:284:osd_idc_find_or_init()) lustre-MDT0000: cannot lookup FID [0x20000e692:0x2:0x0]: rc = -2

Link to test

recovery-small test 111: mdd setup fail should not cause umount oops

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 0 PID: 177918 Comm: ll_mgs_0002 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 37 f8 11 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 24 13 ff 01 0f 84 7b 01 00
RSP: 0018:ff68afa207c73ad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff2adcb76ef1ac00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff2adcb745901300
RBP: ffffffffc132b300 R08: 0000000000000e27 R09: 0000000000000000
R10: ff2adcb76f3a3000 R11: ff2adcb76f3a2dcc R12: ff2adcb77684d0c0
R13: ff2adcb76b382b88 R14: ffffffffc132e310 R15: ff2adcb7466b5b28
FS: 0000000000000000(0000) GS:ff2adcb7bba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000000134210006 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod intel_rapl_msr intel_rapl_common rpcsec_gss_krb5 auth_rpcgss kvm_amd ccp nfsv4 iTCO_wdt dns_resolver iTCO_vendor_support kvm nfs lockd grace fscache irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev i2c_i801 pcspkr lpc_ich virtio_balloon sunrpc ext4 mbcache jbd2 ahci libahci virtio_net crc32c_intel libata serio_raw net_failover failover virtio_blk
CR2: 0000000000000060

Lustre: DEBUG MARKER: lctl set_param fail_loc=0x151
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
LustreError: MGC10.240.23.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail
LustreError: Skipped 2 previous similar messages
Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180
Lustre: Skipped 2 previous similar messages
Lustre: *** cfs_fail_loc=151, val=0***
LustreError: 177910:0:(mdd_device.c:673:mdd_changelog_init()) lustre-MDD0000: changelog setup during init failed: rc = -5
LustreError: 177910:0:(mdd_device.c:1406:mdd_prepare()) lustre-MDD0000: failed to initialize changelog: rc = -5
LustreError: 177910:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -5
Lustre: Failing over lustre-MDT0000
LustreError: 177958:0:(llog_osd.c:225:llog_osd_read_header()) lustre-MDT0001-osp-MDT0000: can't read llog [0x24000040b:0x1:0x0] header: rc = -5
Lustre: 177958:0:(llog_cat.c:847:llog_cat_process_common()) lustre-MDT0001-osp-MDT0000: can't find llog handle [0x24000040b:0x1:0x0]: rc = -5
LustreError: 177958:0:(llog.c:870:llog_process_thread()) lustre-MDT0001-osp-MDT0000 retry remote llog process
LustreError: 177958:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 0, retries 0, failed: rc = -11
Lustre: lustre-MDT0000: Not available for connect from 10.240.23.111@tcp (stopping)
Lustre: Skipped 7 previous similar messages

Link to test

conf-sanity test 5i: start mdt failure at mdt_quota_init()

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 1 PID: 73233 Comm: ll_mgs_0002 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 37 18 13 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 44 14 ff 01 0f 84 7b 01 00
RSP: 0018:ff876f1941b4fad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff4d49442acf3000 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff4d49440436eb80
RBP: ffffffffc143b300 R08: 0000000000000e50 R09: 0000000000000000
R10: ff4d494435c38000 R11: ff4d494435c37d00 R12: ff4d49442a29ec00
R13: ff4d4944276c3b88 R14: ffffffffc143e310 R15: ff4d494401476b28
FS: 0000000000000000(0000) GS:ff4d49447bb00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000000042a10005 CR4: 0000000000771ee0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod intel_rapl_msr intel_rapl_common kvm_amd ccp rpcsec_gss_krb5 auth_rpcgss kvm nfsv4 dns_resolver nfs lockd grace fscache iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i2c_i801 joydev pcspkr lpc_ich virtio_balloon sunrpc ext4 mbcache jbd2 ahci libahci virtio_net libata crc32c_intel serio_raw net_failover failover virtio_blk
CR2: 0000000000000060

Link to test

recovery-small test 107: drop reint reply, then restart MDT

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 0 PID: 163781 Comm: ll_mgs_0000 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 27 b8 11 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e e4 12 ff 01 0f 84 7b 01 00
RSP: 0018:ff611844023ffad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff11e7163871ba00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff11e716033d2a00
RBP: ffffffffc1229300 R08: 0000000000000342 R09: 0000000000000000
R10: ff11e71637ae2000 R11: ff11e71637ae12e7 R12: ff11e7162cecea40
R13: ff11e7163558cb88 R14: ffffffffc122c310 R15: ff11e7160150dc08
FS: 0000000000000000(0000) GS:ff11e7167ba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000000020210003 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? lu_env_refill+0xc/0x50 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? srso_alias_return_thunk+0x5/0xfcdfd
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common dns_resolver kvm_amd ccp nfs lockd grace fscache kvm iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr i2c_i801 virtio_balloon lpc_ich sunrpc ext4 mbcache jbd2 ahci libahci libata crc32c_intel serio_raw virtio_net net_failover virtio_blk failover
CR2: 0000000000000060

Lustre: DEBUG MARKER: lctl set_param fail_loc=0x119
Lustre: *** cfs_fail_loc=119, val=2147483648***
LustreError: 37585:0:(ldlm_lib.c:3332:target_send_reply_msg()) @@@ dropping reply req@ff11e7163750aa40 x1868103326105856/t94489280516(0) o36->a2c32b26-4eda-461a-89ad-c0dfdfab101a@10.240.39.217@tcp:565/0 lens 616/448 e 0 to 0 dl 1781566515 ref 1 fl Interpret:/200/0 rc 0/0 job:'mkdir.0' uid:0 gid:0 projid:4294967295
Lustre: DEBUG MARKER: lctl set_param fail_loc=0
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
Lustre: lustre-MDT0000: Not available for connect from 10.240.39.219@tcp (stopping)
Lustre: Skipped 1 previous similar message

Link to test

conf-sanity test 5h: start mdt failure at mdt_fs_setup()

BUG: unable to handle kernel NULL pointer dereference at 0000000000000060
PGD 0
Oops: 0000 [#1] SMP NOPTI
CPU: 0 PID: 145989 Comm: ll_mgs_0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.125.1.el8_lustre.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
RIP: 0010:qsd_op_end+0x99/0x340 [lquota]
Code: c7 05 0f cd 05 00 01 00 00 00 48 c7 05 0c cd 05 00 00 00 00 00 e8 27 38 51 ff 48 85 db 0f 84 47 02 00 00 48 8b 83 80 00 00 00 <f6> 40 60 02 0f 84 8f 00 00 00 f6 05 9e 64 52 ff 01 0f 84 7b 01 00
RSP: 0018:ff53629701977ad0 EFLAGS: 00010286
RAX: 0000000000000000 RBX: ff3c09aabd335e00 RCX: 0000000000000000
RDX: 0000000000000000 RSI: 0000000000000000 RDI: ff3c09aa82c7e200
RBP: ffffffffc15e6300 R08: 0000000000000ec5 R09: 0000000000000000
R10: ff3c09aabf164000 R11: ff3c09aabf163e6a R12: ff3c09aa8dafd440
R13: ff3c09aaaec61b88 R14: ffffffffc15e9310 R15: ff3c09aa816284d0
FS: 0000000000000000(0000) GS:ff3c09aafba00000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000060 CR3: 0000000139e10006 CR4: 0000000000771ef0
PKRU: 55555554
Call Trace:
? __die_body+0x1a/0x60
? no_context+0x1ba/0x3f0
? __bad_area_nosemaphore+0x157/0x180
? do_page_fault+0x37/0x12d
? page_fault+0x1e/0x30
? qsd_op_end+0x99/0x340 [lquota]
osd_trans_stop+0x363/0xaa0 [osd_ldiskfs]
? srso_alias_return_thunk+0x5/0xfcdfd
? osd_write+0x148/0x700 [osd_ldiskfs]
mgs_nidtbl_write+0x949/0xb20 [mgs]
mgs_ir_update+0x79/0x2a0 [mgs]
mgs_target_reg+0xcd3/0x1eb0 [mgs]
? lustre_pack_reply_flags+0x6a/0x1e0 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? req_capsule_server_pack+0x1e0/0x2b0 [ptlrpc]
? tgt_request_preprocess.isra.34+0x1d2/0x8c0 [ptlrpc]
tgt_request_handle+0x403/0x1d00 [ptlrpc]
ptlrpc_server_handle_request+0x2ca/0xd70 [ptlrpc]
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_add+0x117/0x180 [obdclass]
ptlrpc_main+0xb90/0x1450 [ptlrpc]
? __schedule+0x2d9/0x870
? srso_alias_return_thunk+0x5/0xfcdfd
? ptlrpc_wait_event+0x390/0x390 [ptlrpc]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) ptlrpc_gss(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod intel_rapl_msr intel_rapl_common kvm_amd rpcsec_gss_krb5 auth_rpcgss ccp kvm nfsv4 dns_resolver nfs lockd grace fscache iTCO_wdt iTCO_vendor_support irqbypass crct10dif_pclmul crc32_pclmul joydev ghash_clmulni_intel pcspkr virtio_balloon i2c_i801 lpc_ich sunrpc ext4 mbcache jbd2 ahci libahci virtio_net crc32c_intel serio_raw libata net_failover virtio_blk failover
CR2: 0000000000000060

Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov /dev/mapper/mds3_flakey /mnt/lustre-mds3
LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm183.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}'
Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm275.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm182.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: onyx-156vm182.onyx.whamcloud.com: executing set_default_debug -1 all
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-156vm181.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: DEBUG MARKER: onyx-156vm145.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid
Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true
Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1
Lustre: Failing over lustre-MDT0000
Lustre: server umount lustre-MDT0000 complete
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x80000135
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1
Lustre: DEBUG MARKER: modprobe dm-flakey;
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1
Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1
Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1
LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc
LustreError: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107
LustreError: Skipped 9 previous similar messages
Lustre: *** cfs_fail_loc=135, val=0***
LustreError: 145993:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000 failed (-2)
LustreError: 145993:0:(obd_config.c:2103:class_config_llog_handler()) MGC10.240.24.31@tcp: cfg command failed: rc = -2

Link to test

Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below):
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below):
Limit to a test: (Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Editing crashreport #75039

Added fields:

Failures list (last 100):