| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| recovery-small test 154a: corruption update llog can be skipped | BUG: unable to handle kernel paging request at ffff9d423dab2c1a PGD 12a801067 P4D 12a801067 PUD 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:26890 to 0x2c0000400:27009) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:23976 to 0x280000400:24385) CPU: 0 PID: 112462 Comm: lod0000_rec0001 Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 RIP: 0010:print_llog_hdr+0x568/0x6b0 [obdclass] Code: 46 4a 13 00 48 89 05 47 4a 13 00 c7 05 45 4a 13 00 00 10 00 00 48 c7 05 42 4a 13 00 00 00 00 00 8b 03 48 83 05 38 42 13 00 01 <8b> 54 03 fc e8 6f a5 e0 ff 48 83 05 2f 42 13 00 01 f6 05 e9 14 e2 RSP: 0018:ffffaafe47e93d20 EFLAGS: 00010202 RAX: 00000000dd99ac1e RBX: ffff9d4160118000 RCX: 00000006568f51b4 RDX: 00000006568f51c2 RSI: ffffffffc083765d RDI: ffffffffc08a2440 RBP: ffffffffc081fd28 R08: ffff9d40482c1ce5 R09: 0000000000000000 R10: 000000027925e8dd R11: ffff9d40482c2000 R12: ffffffffc0828b48 R13: ffff9d40522b3000 R14: 0000000000000000 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffff9d4182000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff9d423dab2c1a CR3: 0000000128416003 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_kern_addr_fault+0x73/0x130 ? __do_page_fault+0x8e/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? print_llog_hdr+0x568/0x6b0 [obdclass] lustre_swab_llog_hdr+0x57/0x140 [obdclass] llog_osd_read_header+0x726/0xe90 [obdclass] llog_read_header+0x6a/0x370 [obdclass] llog_init_handle+0xf2/0xc20 [obdclass] lod_sub_prep_llog+0x581/0xcd9 [lod] lod_sub_recovery_thread+0xd6/0xf30 [lod] ? __schedule+0x369/0xcb0 ? do_raw_spin_unlock+0x75/0x190 ? lod_iocontrol+0x7a0/0x7a0 [lod] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: zfs(O) spl(O) lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) ec(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic CR2: ffff9d423dab2c1a | Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 11 previous similar messages LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg314-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg314-server.virtnet: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 20 LustreError: 112462:0:(llog_swab.c:318:lustre_swab_llog_rec()) Unknown llog rec type 0x1069197a swabbing rec ffff9d4160118000 | Link to test |
| recovery-small test 154a: corruption update llog can be skipped | BUG: unable to handle kernel paging request at ffff90744434841b PGD ac601067 P4D ac601067 PUD 0 Oops: 0000 [#1] SMP PTI CPU: 0 PID: 78617 Comm: lod0000_rec0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:print_llog_hdr+0x460/0x4e0 [obdclass] Code: c7 00 55 9e c0 48 89 2d f6 95 0e 00 48 89 05 f7 95 0e 00 c7 05 f5 95 0e 00 00 10 00 00 48 c7 05 f2 95 0e 00 00 00 00 00 8b 03 <8b> 54 03 fc e8 f7 48 da ff f6 05 b9 75 db ff 10 74 5f f6 05 b3 75 RSP: 0018:ffffb60ac871fd20 EFLAGS: 00010202 RAX: 00000000e143041f RBX: ffff907362f18000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffffffffc099bf9c RDI: ffffffffc09e5500 RBP: ffffffffc0984cf8 R08: 0000000000000399 R09: 0000000000000000 R10: ffff907366ba0000 R11: ffff907366b9f398 R12: ffffffffc098d508 R13: ffff907380a25c00 R14: 0000000000000000 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffff9073fcc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff90744434841b CR3: 00000000aac10001 CR4: 00000000000606f0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? print_llog_hdr+0x460/0x4e0 [obdclass] lustre_swab_llog_hdr+0x27/0x100 [obdclass] llog_osd_read_header+0x61c/0xb10 [obdclass] llog_read_header+0x10b/0x2d0 [obdclass] llog_init_handle+0xc7/0x9e0 [obdclass] lod_sub_prep_llog+0x46b/0xa74 [lod] lod_sub_recovery_thread+0xae/0xbb0 [lod] ? __schedule+0x2d9/0x870 ? lod_sub_cancel_llog+0x8e0/0x8e0 [lod] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 Lustre: DEBUG MARKER: onyx-40vm6.onyx.whamcloud.com: executing set_default_debug -1 all ret_from_fork+0x35/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mgs(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon pcspkr joydev i2c_piix4 sunrpc dm_mod ext4 mbcache jbd2 ata_generic ata_piix crc32c_intel libata virtio_net serio_raw virtio_blk net_failover failover CR2: ffff90744434841b | Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-40vm7.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-40vm7.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 4989:0:(mgc_request.c:1854:mgc_process_log()) MGC10.240.23.149@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-40vm6.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-40vm6.onyx.whamcloud.com: executing set_default_debug -1 all | Link to test |
| recovery-small test 154a: corruption update llog can be skipped | BUG: unable to handle kernel paging request at ffffa0ac7338d500 PGD a4e01067 P4D a4e01067 PUD 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 59166 Comm: lod0000_rec0001 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.16.1.el8_lustre.ddn17.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:print_llog_hdr+0x460/0x4e0 [obdclass] Code: c7 80 ae 93 c0 48 89 2d 86 4f 0e 00 48 89 05 87 4f 0e 00 c7 05 85 4f 0e 00 00 10 00 00 48 c7 05 82 4f 0e 00 00 00 00 00 8b 03 <8b> 54 03 fc e8 b7 ba ed ff f6 05 6d 95 f0 ff 10 74 5f f6 05 67 95 RSP: 0018:ffffbb7004f93d30 EFLAGS: 00010202 RAX: 00000000feea5504 RBX: ffffa0ab744e8000 RCX: 0000000000000000 RDX: 0000000000000169 RSI: ffffffffc08f1976 RDI: ffffffffc093ae80 RBP: ffffffffc08db868 R08: 000000000000000f R09: 0000000000000000 R10: ffffa0ab770d7000 R11: ffffa0ab770d6190 R12: ffffffffc08e55e0 R13: ffffa0ab45961400 R14: 0000000000000000 R15: 0000000000000002 FS: 0000000000000000(0000) GS:ffffa0abffd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffffa0ac7338d500 CR3: 00000000a3410001 CR4: 00000000000606e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? print_llog_hdr+0x460/0x4e0 [obdclass] lustre_swab_llog_hdr+0x37/0x100 [obdclass] llog_osd_read_header+0x996/0x9a0 [obdclass] llog_read_header+0xfd/0x290 [obdclass] llog_init_handle+0xc7/0x9a0 [obdclass] ? llog_open+0x220/0x440 [obdclass] lod_sub_prep_llog+0x367/0x9f3 [lod] lod_sub_recovery_thread+0xed/0xb50 [lod] ? __schedule+0x2d9/0x870 ? lod_sub_cancel_llog+0x890/0x890 [lod] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) dm_flakey rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon joydev i2c_piix4 pcspkr sunrpc dm_mod ext4 mbcache jbd2 ata_generic ata_piix crc32c_intel virtio_net serio_raw libata virtio_blk net_failover failover CR2: ffffa0ac7338d500 | LustreError: 5880:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 10.240.40.254@tcp arrived at 1726925360 with bad export cookie 10460289095229997894 Lustre: lustre-MDT0000-lwp-MDT0002: Connection restored to (at 0@lo) Lustre: Skipped 13 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-58vm7.trevis.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: trevis-58vm7.trevis.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 LustreError: 58240:0:(ldlm_lib.c:2946:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery Lustre: 56814:0:(ldlm_lib.c:2349:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: 56814:0:(ldlm_lib.c:2349:target_recovery_overseer()) Skipped 2 previous similar messages LustreError: 56814:0:(ldlm_lib.c:1883:abort_lock_replay_queue()) @@@ aborted: req@00000000593687e2 x1810804502392064/t0(0) o101->lustre-MDT0003-mdtlov_UUID@10.240.40.254@tcp:294/0 lens 328/0 e 1 to 0 dl 1726925384 ref 1 fl Complete:/40/ffffffff rc 0/-1 job:'ldlm_lock_repla.0' Lustre: lustre-MDT0000-osd: cancel update llog [0x20000fe00:0x1:0x0] Lustre: lustre-MDT0000: Not available for connect from 10.240.40.254@tcp (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x24000040a:0x1:0x0] LustreError: 56814:0:(client.c:1278:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008363d1fe x1810811727587264/t0(0) o700->lustre-MDT0001-osp-MDT0000@10.240.40.254@tcp:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/0/ffffffff rc 0/-1 job:'tgt_recover_0.0' LustreError: 56814:0:(client.c:1278:ptlrpc_import_delay_req()) Skipped 2 previous similar messages LustreError: 56814:0:(fid_request.c:234:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 56814:0:(fid_request.c:234:seq_client_alloc_seq()) Skipped 2 previous similar messages LustreError: 56814:0:(fid_request.c:396:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 56814:0:(fid_request.c:396:seq_client_alloc_fid()) Skipped 2 previous similar messages Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x280000409:0x1:0x0] Lustre: lustre-MDT0003-osp-MDT0000: cancel update llog [0x2c000040a:0x1:0x0] Lustre: lustre-MDT0000: Recovery over after 0:13, of 5 clients 0 recovered and 5 were evicted. Lustre: 56814:0:(mdt_handler.c:7739:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-58vm6.trevis.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-58vm6.trevis.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: trevis-58vm6.trevis.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: trevis-58vm6.trevis.whamcloud.com: executing set_default_debug -1 all 4 Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-58vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 20 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-58vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 20 Lustre: DEBUG MARKER: trevis-58vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 20 Lustre: DEBUG MARKER: trevis-58vm6.trevis.whamcloud.com: executing _wait_recovery_complete *.lustre-MDT0000.recovery_status 20 LustreError: 59166:0:(llog_swab.c:339:lustre_swab_llog_rec()) Unknown llog rec type 0x1069099a swabbing rec 000000007e6d3df7 | Link to test |
| recovery-small test 154a: corruption update llog can be skipped | BUG: unable to handle kernel paging request at ffff88c6e5672c65 PGD 58201067 P4D 58201067 PUD 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 90770 Comm: lod0000_rec0001 Kdump: loaded Tainted: G OE --------- - - 4.18.0-513.18.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:print_llog_hdr+0x460/0x4e0 [obdclass] Code: c7 60 d1 91 c0 48 89 2d 76 ab 0d 00 48 89 05 77 ab 0d 00 c7 05 75 ab 0d 00 00 10 00 00 48 c7 05 72 ab 0d 00 00 00 00 00 8b 03 <8b> 54 03 fc e8 b7 fe da ff f6 05 d9 ae dc ff 10 74 5f f6 05 d3 ae RSP: 0018:ffffb191465e3d20 EFLAGS: 00010202 RAX: 00000000aa66ac69 RBX: ffff88c63b008000 RCX: 0000000000000000 RDX: 0000000000000239 RSI: ffffffffc08d7eed RDI: ffffffffc091d160 RBP: ffffffffc08c2948 R08: 0000000000000261 R09: 0000000000000000 R10: ffff88c643bd9000 R11: ffff88c643bd8260 R12: ffffffffc08ca658 R13: ffff88c642324380 R14: 0000000000000002 R15: 0000000000000000 FS: 0000000000000000(0000) GS:ffff88c6bfd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: ffff88c6e5672c65 CR3: 0000000055c10005 CR4: 00000000001706e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x16c/0x1c0 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? print_llog_hdr+0x460/0x4e0 [obdclass] lustre_swab_llog_hdr+0x27/0x100 [obdclass] llog_osd_read_header+0x538/0xa30 [obdclass] llog_read_header+0x10b/0x2c0 [obdclass] llog_init_handle+0xc7/0x9e0 [obdclass] ? llog_open+0x229/0x450 [obdclass] lod_sub_prep_llog+0x46b/0xb15 [lod] lod_sub_recovery_thread+0xae/0xb70 [lod] ? __schedule+0x2d9/0x870 ? lod_sub_cancel_llog+0x8e0/0x8e0 [lod] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mgs(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) dm_flakey ptlrpc_gss(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) libcfs(OE) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i2c_piix4 pcspkr virtio_balloon joydev sunrpc dm_mod ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net serio_raw net_failover failover virtio_blk CR2: ffff88c6e5672c65 | Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-70vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov,skpath=/tmp/test-framework-keys /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-70vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-70vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u | Link to test |