| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| sanity-lfsck test 4: FID-in-dirent can be rebuilt after MDT file-level backup/restore | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 284251 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.76.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x23f/0x4e0 [osd_ldiskfs] Code: 8b 36 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 71 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffffb0f909877c40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8fc642860000 RCX: 0000000000000003 RDX: 0000000000000058 RSI: 0000000000000001 RDI: 0000000000000000 RBP: ffff8fc635255400 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc075a300 R11: ffff8fc654b60a10 R12: 0000000000000000 R13: 0000000000000000 R14: ffff8fc654b60008 R15: 000000000000010e FS: 0000000000000000(0000) GS:ffff8fc6bfd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000049410002 CR4: 00000000001706e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x23f/0x4e0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3ab/0xc10 [osd_ldiskfs] ? osd_iit_iget+0x4e0/0x4e0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? lfsck_needs_scan_dir+0x2d/0xcb0 [lfsck] ? fld_cache_lookup+0x31/0x1c0 [fld] lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr nfs lockd grace fscache intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon joydev pcspkr i2c_piix4 sunrpc ext4 mbcache ata_generic jbd2 ata_piix libata virtio_net net_failover crc32c_intel serio_raw virtio_blk failover [last unloaded: dm_flakey] CR2: 0000000000000000 | Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 LustreError: 280473:0:(obd_class.h:479:obd_check_dev()) Device 14 not setup LustreError: 280473:0:(obd_class.h:479:obd_check_dev()) Skipped 245 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null) Lustre: DEBUG MARKER: tar zcf /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt/ . Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt Lustre: DEBUG MARKER: [ -e /dev/mapper/mds1_flakey ] Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: mkfs.lustre --mgs --fsname=lustre --mdt --index=0 --param=sys.timeout=20 --param=mdt.identity_upcall=/usr/sbin/l_getidentity --backfstype=ldiskfs --device-size=100000 --mkfsoptions="-b 4096" --reformat /dev/mapper/mds1_flakey LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro Lustre: DEBUG MARKER: mount -t ldiskfs /dev/mapper/mds1_flakey /mnt/lustre-brpt LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null) Lustre: DEBUG MARKER: tar zxfp /tmp/backup_restore.tgz --xattrs --xattrs-include=trusted.* --sparse -C /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -fv /mnt/lustre-brpt/OBJECTS/* /mnt/lustre-brpt/CATALOGS Lustre: DEBUG MARKER: umount -d /mnt/lustre-brpt Lustre: DEBUG MARKER: rm -f /tmp/backup_restore.tgz Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey lustre-MDT0000 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov -o user_xattr,noscrub /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc Lustre: lustre-MDT0000: reset Object Index mappings LustreError: MGC10.240.29.74@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 9 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-113vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-113vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-113vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-113vm7.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=1 fail_loc=0x1601 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -r LustreError: 284251:0:(lfsck_engine.c:1045:lfsck_master_engine()) lustre-MDT0000-osd: master engine fail to verify the .lustre/lost+found/, go ahead: rc = -115 | Link to test |
| sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 140880 Comm: lfsck Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 RIP: 0010:osd_iit_iget+0x67c/0x720 [osd_ldiskfs] Code: ef fe 48 c7 c7 80 53 7e c1 48 83 05 25 3f 06 00 01 e8 e8 87 ef fe 48 83 05 88 4a 06 00 01 e9 eb fb ff ff 4c 8b 83 28 79 00 00 <49> 3b 38 74 4b 48 8b 05 d0 4a 06 00 48 8d 48 01 31 c0 eb 11 48 63 RSP: 0018:ffffc238ca04fc40 EFLAGS: 00010246 RAX: 0000000000000002 RBX: ffff9e45c6e38000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000200000005 RBP: ffff9e45d175c4d8 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: ffff9e44cb1c8000 R12: 0000000000000002 R13: 0000000000000000 R14: ffff9e45efa722d8 R15: 0000000000004e53 FS: 0000000000000000(0000) GS:ffff9e4602000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000003b616001 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? xattr_find_entry+0xd7/0x180 [ldiskfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? osd_iit_iget+0x67c/0x720 [osd_ldiskfs] osd_preload_next+0xa5/0xe0 [osd_ldiskfs] osd_inode_iteration+0x470/0x1150 [osd_ldiskfs] ? osd_iit_iget+0x720/0x720 [osd_ldiskfs] ? osd_obj_spec_lookup+0x3d0/0x3d0 [osd_ldiskfs] ? __wake_up_common_lock+0x9c/0x130 osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] ? lu_object_put+0x223/0x550 [obdclass] lfsck_master_oit_engine+0x3aa/0x1b90 [lfsck] lfsck_master_engine+0x623/0x12f0 [lfsck] ? __schedule+0x369/0xcb0 ? woken_wake_function+0x30/0x30 ? lfsck_prep+0xd70/0xd70 [lfsck] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey zfs(O) spl(O) ec(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl i2c_piix4 pcspkr squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: 0000000000000000 | Link to test | |
| sanity-scrub test 11: OI scrub skips the new created objects only once | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 262141 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.71.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x23f/0x4e0 [osd_ldiskfs] Code: 8b 36 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 71 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffff9ee205cbbc40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8dfa781f0000 RCX: 0000000000000003 RDX: 0000000000000018 RSI: 0000000200000003 RDI: 0000000000000000 RBP: ffff8dfa7e270f40 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc10a8660 R11: ffff8dfa7e526a10 R12: 0000000000000002 R13: 0000000000000000 R14: ffff8dfa7e526378 R15: 0000000000007d40 FS: 0000000000000000(0000) GS:ffff8dfaffd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 00000000a4610003 CR4: 00000000001706e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x23f/0x4e0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3ab/0xc10 [osd_ldiskfs] ? osd_iit_iget+0x4e0/0x4e0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] ? __schedule+0x2d9/0x870 ? prepare_to_wait_event+0xa0/0x180 osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) osd_ldiskfs(OE) lquota(OE) ldiskfs(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey ec(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel i2c_piix4 pcspkr joydev virtio_balloon sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net net_failover serio_raw virtio_blk failover [last unloaded: libcfs] CR2: 0000000000000000 | Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-96vm7.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-96vm6.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-96vm6.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-96vm3.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-96vm2.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-96vm7.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-96vm6.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-96vm3.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-96vm6.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: trevis-96vm2.trevis.whamcloud.com: executing load_modules_local Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov /dev/mapper/mds3_flakey /mnt/lustre-mds3 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: 260487:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 Lustre: 260487:0:(mgs_llog.c:1345:mgs_modify_param()) Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-96vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-96vm2.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 3 -r Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 3 -r Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub | Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub | Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0002.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 6 -r Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0002 -t scrub 6 -r | Link to test |
| sanity-scrub test 9: OI scrub speed control | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 1 PID: 113842 Comm: lfsck Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.2-1.fc38 04/01/2014 RIP: 0010:osd_iit_iget+0x67c/0x720 [osd_ldiskfs] Code: f1 fe 48 c7 c7 80 33 7e c1 48 83 05 25 3f 06 00 01 e8 e8 d7 f0 fe 48 83 05 88 4a 06 00 01 e9 eb fb ff ff 4c 8b 83 28 79 00 00 <49> 3b 38 74 4b 48 8b 05 d0 4a 06 00 48 8d 48 01 31 c0 eb 11 48 63 RSP: 0018:ffffae4189edfc40 EFLAGS: 00010246 RAX: 0000000000000002 RBX: ffff9c25ea368000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000000000001 RBP: ffff9c24c1616998 R08: 0000000000000000 R09: 0000000000000000 R10: ffffffffc1649450 R11: ffffffffc17b9349 R12: 0000000000000002 R13: 0000000000000000 R14: ffff9c24cbe96418 R15: 0000000000007d33 FS: 0000000000000000(0000) GS:ffff9c2602080000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000004da16005 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? xattr_find_entry+0xd7/0x180 [ldiskfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x40/0x40 [ldiskfs] ? osd_iit_iget+0x67c/0x720 [osd_ldiskfs] osd_preload_next+0xa5/0xe0 [osd_ldiskfs] osd_inode_iteration+0x470/0x1150 [osd_ldiskfs] ? osd_iit_iget+0x720/0x720 [osd_ldiskfs] ? osd_obj_spec_lookup+0x3d0/0x3d0 [osd_ldiskfs] ? __wake_up_common_lock+0x9c/0x130 osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] ? lu_object_put+0x223/0x550 [obdclass] lfsck_master_oit_engine+0x3aa/0x1b90 [lfsck] lfsck_master_engine+0x623/0x12f0 [lfsck] ? _raw_spin_unlock_irq+0x12/0x50 ? __schedule+0xa4d/0xcb0 ? woken_wake_function+0x30/0x30 ? lfsck_prep+0xd70/0xd70 [lfsck] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey zfs(O) spl(O) ec(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs ata_generic crct10dif_pclmul crc32_pclmul crc32c_intel ata_piix ghash_clmulni_intel serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: 0000000000000000 | Lustre: DEBUG MARKER: oleg222-server.virtnet: executing load_modules_local Lustre: 108188:0:(mgs_llog.c:1345:mgs_modify_param()) MGS: modify general/debug_raw_pointers=Y (mode = 0) failed: rc = -17 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 32 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 LustreError: 110091:0:(obd_class.h:479:obd_check_dev()) Device 27 not setup LustreError: 110091:0:(obd_class.h:479:obd_check_dev()) Skipped 71 previous similar messages Lustre: server umount lustre-MDT0001 complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) Lustre: 16274:0:(client.c:2470:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1757497176/real 1757497176] req@ffff9c24c2f0c380 x1842866887611776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1757497192 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: 16274:0:(client.c:2470:ptlrpc_expire_one_request()) Skipped 56 previous similar messages LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: (null) LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,user_xattr,no_mbcache,nodelalloc Lustre: lustre-MDT0000: reset Object Index mappings Lustre: Skipped 3 previous similar messages LustreError: 16271:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff9c24c2f0e300 x1842866887615104/t0(0) o250->MGC192.168.202.122@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:394 to 0x280000400:417) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:394 to 0x2c0000400:417) Lustre: DEBUG MARKER: oleg222-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:394 to 0x280000401:417) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:394 to 0x2c0000401:417) Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) Lustre: Skipped 22 previous similar messages | Link to test |
| sanity-lfsck test 23a: LFSCK can repair dangling name entry (1) | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 1 PID: 132082 Comm: lfsck Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 RIP: 0010:osd_iit_iget+0x67c/0x720 [osd_ldiskfs] Code: f4 fe 48 c7 c7 40 a3 91 c1 48 83 05 b5 2e 06 00 01 e8 d8 47 f4 fe 48 83 05 18 3a 06 00 01 e9 eb fb ff ff 4c 8b 83 28 79 00 00 <49> 3b 38 74 4b 48 8b 05 60 3a 06 00 48 8d 48 01 31 c0 eb 11 48 63 RSP: 0018:ffffb38ccf283c40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8a843dd38000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000200000003 RBP: ffff8a842a6313d8 R08: 0000000000000000 R09: 0000000000000000 R10: ffffffffc1781c00 R11: ffffffffc18f131c R12: 0000000000000002 R13: 0000000000000000 R14: ffff8a83096882b0 R15: 0000000000004e7d FS: 0000000000000000(0000) GS:ffff8a8442080000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000067216003 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? xattr_find_entry+0xd7/0x180 [ldiskfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x40/0x40 [ldiskfs] ? osd_iit_iget+0x67c/0x720 [osd_ldiskfs] osd_preload_next+0xa5/0xe0 [osd_ldiskfs] osd_inode_iteration+0x470/0x1150 [osd_ldiskfs] ? osd_iit_iget+0x720/0x720 [osd_ldiskfs] ? osd_obj_spec_lookup+0x3d0/0x3d0 [osd_ldiskfs] ? do_raw_spin_unlock+0x75/0x190 ? do_raw_spin_unlock+0x75/0x190 osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x3aa/0x1b90 [lfsck] lfsck_master_engine+0x623/0x12f0 [lfsck] ? __schedule+0x369/0xcb0 ? woken_wake_function+0x30/0x30 ? lfsck_prep+0xd70/0xd70 [lfsck] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: 0000000000000000 | Lustre: *** cfs_fail_loc=1620, val=0*** | Link to test |
| sanity-lfsck test 31a: The LFSCK can find/repair the name entry with bad name hash (1) | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 141592 Comm: lfsck Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 RIP: 0010:osd_iit_iget+0x67c/0x720 [osd_ldiskfs] Code: f5 fe 48 c7 c7 80 c3 86 c1 48 83 05 35 3c 06 00 01 e8 d8 44 f5 fe 48 83 05 98 47 06 00 01 e9 eb fb ff ff 4c 8b 83 28 79 00 00 <49> 3b 38 74 4b 48 8b 05 e0 47 06 00 48 8d 48 01 31 c0 eb 11 48 63 RSP: 0018:ffffbf0949df3c40 EFLAGS: 00010246 RAX: 0000000000000002 RBX: ffff99472e8d0000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000200000005 RBP: ffff994727f3e378 R08: 0000000000000000 R09: 0000000000000000 R10: ffffffffc16d2c00 R11: ffffffffc184338c R12: 0000000000000002 R13: 0000000000000000 R14: ffff994706eba5a8 R15: 0000000000004e4b FS: 0000000000000000(0000) GS:ffff994742000000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000109a16003 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? xattr_find_entry+0xd7/0x180 [ldiskfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x40/0x40 [ldiskfs] ? osd_iit_iget+0x67c/0x720 [osd_ldiskfs] osd_preload_next+0xa5/0xe0 [osd_ldiskfs] osd_inode_iteration+0x470/0x1150 [osd_ldiskfs] ? osd_iit_iget+0x720/0x720 [osd_ldiskfs] ? osd_obj_spec_lookup+0x3d0/0x3d0 [osd_ldiskfs] ? __wake_up_common_lock+0x9c/0x130 osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] ? lu_object_put+0x223/0x550 [obdclass] lfsck_master_oit_engine+0x3aa/0x1b90 [lfsck] lfsck_master_engine+0x623/0x12f0 [lfsck] ? __schedule+0x369/0xcb0 ? woken_wake_function+0x30/0x30 ? lfsck_prep+0xd70/0xd70 [lfsck] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: 0000000000000000 | Link to test | |
| conf-sanity test 61a: large xattr | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 1 PID: 368587 Comm: lfsck Kdump: loaded Tainted: G O -------- - - 4.18.0rh8.10-debug #2 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-2.fc40 04/01/2014 RIP: 0010:osd_iit_iget+0x67c/0x720 [osd_ldiskfs] Code: f4 fe 48 c7 c7 40 d3 5d c1 48 83 05 45 3c 06 00 01 e8 68 c5 f4 fe 48 83 05 a8 47 06 00 01 e9 eb fb ff ff 4c 8b 83 28 79 00 00 <49> 3b 38 74 4b 48 8b 05 f0 47 06 00 48 8d 48 01 31 c0 eb 11 48 63 RSP: 0018:ffffad8609e77c40 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff930909820000 RCX: 0000000000000000 RDX: 000000000000000f RSI: 0000000000000003 RDI: 0000000200000003 RBP: ffff930907aff5d8 R08: 0000000000000000 R09: 0000000000000000 R10: ffffffffc1443c00 R11: ffffffffc15b433c R12: 0000000000000000 R13: 0000000000000000 R14: ffff930815b524e0 R15: 000000000000d085 FS: 0000000000000000(0000) GS:ffff930942080000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000115816003 CR4: 0000000000170ee0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a ? no_context+0x30f/0x5a0 ? xattr_find_entry+0xd7/0x180 [ldiskfs] ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x30f ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x40/0x40 [ldiskfs] ? osd_iit_iget+0x67c/0x720 [osd_ldiskfs] osd_preload_next+0xa5/0xe0 [osd_ldiskfs] osd_inode_iteration+0x470/0x1150 [osd_ldiskfs] ? osd_iit_iget+0x720/0x720 [osd_ldiskfs] ? osd_obj_spec_lookup+0x3d0/0x3d0 [osd_ldiskfs] osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] ? refcount_dec_and_test+0x15/0x20 ? lfsck_master_dir_engine+0x8da/0xd90 [lfsck] lfsck_master_oit_engine+0x3aa/0x1b90 [lfsck] lfsck_master_engine+0x623/0x12f0 [lfsck] ? _raw_spin_unlock_irqrestore+0x4f/0x90 ? lfsck_prep+0xd70/0xd70 [lfsck] kthread+0x1d1/0x200 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey loop zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul crc32c_intel ata_generic ghash_clmulni_intel ata_piix serio_raw libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: 0000000000000000 | LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 352627:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 352627:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS Lustre: Failing over lustre-MDT0000 LustreError: 355069:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup LustreError: 355069:0:(obd_class.h:479:obd_check_dev()) Skipped 13 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 352622:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Skipped 2 previous similar messages LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: 352622:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 2 previous similar messages LustreError: 352627:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 352627:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 3 previous similar messages LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity Lustre: Failing over lustre-MDT0000 LustreError: 356142:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup LustreError: 356142:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: 352622:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 352622:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 1 previous similar message Lustre: Skipped 1 previous similar message LustreError: 352627:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 352627:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 352609:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752661919 with bad export cookie 10294851625259161027 LustreError: 352609:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete LustreError: 357566:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup LustreError: 357566:0:(obd_class.h:479:obd_check_dev()) Skipped 17 previous similar messages Lustre: server umount lustre-OST0000 complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 358394:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 358394:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? Lustre: server umount lustre-MDT0000 complete LustreError: 358375:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752661991 with bad export cookie 10294851625259161734 LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 358375:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: DEBUG MARKER: oleg118-server.virtnet: executing unload_modules_local Key type lgssc unregistered LNet: 361572:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.201.118@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: oleg118-server.virtnet: executing load_modules_local libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.16.57 LNet: Added LNI 192.168.201.118@tcp [8/256/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 365286:0:(obd_class.h:479:obd_check_dev()) Device 9 not setup Lustre: server umount lustre-MDT0000 complete LustreError: 363034:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 363034:0:(ldlm_lib.c:1113:target_handle_connect()) Skipped 4 previous similar messages LustreError: 363019:0:(ldlm_lockd.c:2550:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1752662057 with bad export cookie 10617238526864635357 LustreError: MGC192.168.201.118@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 365490:0:(obd_class.h:479:obd_check_dev()) Device 16 not setup LustreError: 365490:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 365690:0:(obd_class.h:479:obd_check_dev()) Device 2 not setup LustreError: 365690:0:(obd_class.h:479:obd_check_dev()) Skipped 5 previous similar messages Lustre: server umount lustre-OST0000 complete LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 366418:0:(ldlm_lib.c:1113:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg118-server.virtnet: executing set_default_debug -1 all Lustre: DEBUG MARKER: oleg118-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) | Link to test |
| conf-sanity test 108b: migrate from ZFS to ldiskfs | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 1900131 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffffb07301dcbc38 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff9c2155ff0000 RCX: 0000000000000003 RDX: 0000000000000018 RSI: 0000000200000003 RDI: 0000000000000000 RBP: ffff9c214982a700 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc1436860 R11: ffff9c212c38a39c R12: 000000000000d08c R13: 0000000000000002 R14: 0000000000000000 R15: ffff9c21340619e0 FS: 0000000000000000(0000) GS:ffff9c21bfd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000067e10005 CR4: 00000000001706e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] ? osd_iit_iget+0x1f9/0x4f0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4f0/0x4f0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: ofd(OE) osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lustre(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey nfsv3 nfs_acl loop dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel virtio_balloon joydev pcspkr i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata virtio_net crc32c_intel serio_raw net_failover failover virtio_blk [last unloaded: libcfs] CR2: 0000000000000000 | Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: rm -rf /tmp/d108b.conf-sanity Lustre: DEBUG MARKER: mkdir -p /tmp/d108b.conf-sanity/mnt /tmp/d108b.conf-sanity/images Lustre: DEBUG MARKER: mkdir /tmp/d108b.conf-sanity/mnt/mdt1 Lustre: DEBUG MARKER: dd if=/dev/zero of=/tmp/d108b.conf-sanity/images/mdt1 seek=199 bs=1M count=1 Lustre: DEBUG MARKER: mkdir /tmp/d108b.conf-sanity/mnt/mdt2 Lustre: DEBUG MARKER: dd if=/dev/zero of=/tmp/d108b.conf-sanity/images/mdt2 seek=199 bs=1M count=1 Lustre: DEBUG MARKER: mkdir /tmp/d108b.conf-sanity/mnt/ost1 Lustre: DEBUG MARKER: dd if=/dev/zero of=/tmp/d108b.conf-sanity/images/ost1 seek=199 bs=1M count=1 Lustre: DEBUG MARKER: mkdir /tmp/d108b.conf-sanity/mnt/ost2 Lustre: DEBUG MARKER: dd if=/dev/zero of=/tmp/d108b.conf-sanity/images/ost2 seek=199 bs=1M count=1 Lustre: DEBUG MARKER: mkfs.lustre --fsname=lustre --mgs --mdt --index=0 --replace --backfstype=ldiskfs --device-size=200000 --reformat /tmp/d108b.conf-sanity/images/mdt1 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro Lustre: DEBUG MARKER: mkfs.lustre --fsname=lustre --mgsnode=10.240.28.148@tcp --mdt --index=1 --replace --backfstype=ldiskfs --device-size=200000 --reformat /tmp/d108b.conf-sanity/images/mdt2 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro Lustre: DEBUG MARKER: mkfs.lustre --fsname=lustre --mgsnode=10.240.28.148@tcp --ost --index=0 --replace --backfstype=ldiskfs --device-size=200000 --reformat /tmp/d108b.conf-sanity/images/ost1 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro Lustre: DEBUG MARKER: mkfs.lustre --fsname=lustre --mgsnode=10.240.28.148@tcp --ost --index=1 --replace --backfstype=ldiskfs --device-size=200000 --reformat /tmp/d108b.conf-sanity/images/ost2 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro Lustre: DEBUG MARKER: mount -t ldiskfs -o loop /tmp/d108b.conf-sanity/images/mdt1 /tmp/d108b.conf-sanity/mnt/mdt1 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Lustre: DEBUG MARKER: tar jxf /usr/lib64/lustre/tests/zfs_mdt1_2_11.tar.bz2 --xattrs --xattrs-include=*.* -C /tmp/d108b.conf-sanity/mnt/mdt1/ Lustre: DEBUG MARKER: cd /tmp/d108b.conf-sanity/mnt/mdt1 && rm -rf oi.* OI_* lfsck_* LFSCK Lustre: DEBUG MARKER: umount /tmp/d108b.conf-sanity/mnt/mdt1 Lustre: DEBUG MARKER: mount -t ldiskfs -o loop /tmp/d108b.conf-sanity/images/mdt2 /tmp/d108b.conf-sanity/mnt/mdt2 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Lustre: DEBUG MARKER: tar jxf /usr/lib64/lustre/tests/zfs_mdt2_2_11.tar.bz2 --xattrs --xattrs-include=*.* -C /tmp/d108b.conf-sanity/mnt/mdt2/ Lustre: DEBUG MARKER: cd /tmp/d108b.conf-sanity/mnt/mdt2 && rm -rf oi.* OI_* lfsck_* LFSCK Lustre: DEBUG MARKER: umount /tmp/d108b.conf-sanity/mnt/mdt2 Lustre: DEBUG MARKER: mount -t ldiskfs -o loop /tmp/d108b.conf-sanity/images/ost1 /tmp/d108b.conf-sanity/mnt/ost1 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Lustre: DEBUG MARKER: tar jxf /usr/lib64/lustre/tests/zfs_ost1_2_11.tar.bz2 --xattrs --xattrs-include=*.* -C /tmp/d108b.conf-sanity/mnt/ost1/ Lustre: DEBUG MARKER: cd /tmp/d108b.conf-sanity/mnt/ost1 && rm -rf oi.* OI_* lfsck_* LFSCK Lustre: DEBUG MARKER: umount /tmp/d108b.conf-sanity/mnt/ost1 Lustre: DEBUG MARKER: mount -t ldiskfs -o loop /tmp/d108b.conf-sanity/images/ost2 /tmp/d108b.conf-sanity/mnt/ost2 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: (null) Lustre: DEBUG MARKER: tar jxf /usr/lib64/lustre/tests/zfs_ost2_2_11.tar.bz2 --xattrs --xattrs-include=*.* -C /tmp/d108b.conf-sanity/mnt/ost2/ Lustre: DEBUG MARKER: cd /tmp/d108b.conf-sanity/mnt/ost2 && rm -rf oi.* OI_* lfsck_* LFSCK Lustre: DEBUG MARKER: umount /tmp/d108b.conf-sanity/mnt/ost2 Lustre: DEBUG MARKER: mount -t lustre -o nosvc,loop /tmp/d108b.conf-sanity/images/mdt1 /tmp/d108b.conf-sanity/mnt/mdt1 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index 'nodemap' with [0x200000003:0x3:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000-MDT0000' with [0x200000005:0x2:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000003:0xa:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000-MDT0000' with [0x200000005:0x3:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000003:0xc:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000003:0xb:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000-MDT0000' with [0x200000005:0x1:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2020000' with [0x200000006:0x2020000:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1020000' with [0x200000006:0x1020000:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x20000' with [0x200000006:0x20000:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x10000' with [0x200000006:0x10000:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x2010000' with [0x200000006:0x2010000:0x0]: rc = 0 Lustre: 1898861:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0000: restore index '0x1010000' with [0x200000006:0x1010000:0x0]: rc = 0 Lustre: DEBUG MARKER: lctl replace_nids lustre-MDT0000 10.240.28.148@tcp Lustre: DEBUG MARKER: lctl replace_nids lustre-MDT0001 10.240.28.148@tcp Lustre: DEBUG MARKER: lctl replace_nids lustre-OST0000 10.240.28.148@tcp Lustre: DEBUG MARKER: lctl replace_nids lustre-OST0001 10.240.28.148@tcp Lustre: DEBUG MARKER: umount /tmp/d108b.conf-sanity/mnt/mdt1 Lustre: server umount MGS complete Lustre: DEBUG MARKER: mount -t lustre -o loop,abort_recov /tmp/d108b.conf-sanity/images/mdt1 /tmp/d108b.conf-sanity/mnt/mdt1 loop0: detected capacity change from 0 to 204800000 LDISKFS-fs (loop0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Aborting client recovery Lustre: DEBUG MARKER: mount -t lustre -o loop,abort_recov /tmp/d108b.conf-sanity/images/mdt2 /tmp/d108b.conf-sanity/mnt/mdt2 loop1: detected capacity change from 0 to 204800000 LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000-MDT0001' with [0x200000005:0x2:0x0]: rc = 0 Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000-MDT0001' with [0x200000005:0x1:0x0]: rc = 0 Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x10000' with [0x200000003:0x4:0x0]: rc = 0 Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x1010000' with [0x200000003:0x5:0x0]: rc = 0 Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000' with [0x200000003:0x6:0x0]: rc = 0 Lustre: 1899632:0:(osd_scrub.c:1842:osd_index_restore()) lustre-MDT0001: restore index '0x2010000-MDT0001' with [0x200000005:0x3:0x0]: rc = 0 Lustre: Found index 1 for lustre-MDT0001, updating log Lustre: 1899632:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC10.240.28.148@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: lustre-MDT0001: Aborting client recovery Lustre: DEBUG MARKER: mount -t lustre -o loop,abort_recov /tmp/d108b.conf-sanity/images/ost1 /tmp/d108b.conf-sanity/mnt/ost1 loop2: detected capacity change from 0 to 204800000 LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000-OST0000' with [0x200000005:0x1:0x0]: rc = 0 Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000-OST0000' with [0x200000005:0x2:0x0]: rc = 0 Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x2020000-OST0000' with [0x200000005:0x3:0x0]: rc = 0 Lustre: 1899763:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0000: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 Lustre: Found index 0 for lustre-OST0000, updating log LustreError: 1899528:0:(lod_dev.c:510:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 3, retries 0, failed: rc = -108 LustreError: 1899528:0:(lod_dev.c:510:lod_sub_recovery_thread()) Skipped 2 previous similar messages Lustre: lustre-OST0000: Aborting recovery Lustre: DEBUG MARKER: mount -t lustre -o loop,abort_recov /tmp/d108b.conf-sanity/images/ost2 /tmp/d108b.conf-sanity/mnt/ost2 loop3: detected capacity change from 0 to 204800000 LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: errors=remount-ro LDISKFS-fs (loop3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index 'fld' with [0x200000001:0x3:0x0]: rc = 0 Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000-OST0001' with [0x200000005:0x1:0x0]: rc = 0 Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000' with [0x200000003:0x6:0x0]: rc = 0 Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000' with [0x200000003:0x7:0x0]: rc = 0 Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x20000' with [0x200000003:0x5:0x0]: rc = 0 Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x1020000-OST0001' with [0x200000005:0x2:0x0]: rc = 0 Lustre: 1899902:0:(osd_scrub.c:1842:osd_index_restore()) lustre-OST0001: restore index '0x2020000-OST0001' with [0x200000005:0x3:0x0]: rc = 0 Lustre: Found index 1 for lustre-OST0001, updating log Lustre: lustre-OST0001: Aborting recovery Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0001 -t scrub | Link to test |
| conf-sanity test 61b: large xattr | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 1261249 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffffa3670886fc38 EFLAGS: 00010246 RAX: 0000000000000002 RBX: ffff8b2533048000 RCX: 0000000000000003 RDX: 0000000000000018 RSI: 0000000200000005 RDI: 0000000000000000 RBP: ffff8b252564abc0 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc0678860 R11: ffff8b25327246c8 R12: 000000000000d066 R13: 0000000000000002 R14: 0000000000000000 R15: ffff8b2502db1418 FS: 0000000000000000(0000) GS:ffff8b25bfd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000084610006 CR4: 00000000001706e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] ? osd_iit_iget+0x1f9/0x4f0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4f0/0x4f0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_flakey nfsv3 nfs_acl loop dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common crct10dif_pclmul dns_resolver nfs crc32_pclmul lockd grace fscache ghash_clmulni_intel joydev pcspkr virtio_balloon i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata virtio_net virtio_blk crc32c_intel net_failover serio_raw failover [last unloaded: libcfs] CR2: 0000000000000000 | Lustre: DEBUG MARKER: dumpe2fs -h /dev/mapper/mds1_flakey 2>&1 | Lustre: DEBUG MARKER: dumpe2fs -h /dev/mapper/mds1_flakey 2>&1 | Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm3.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm3.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 Lustre: lustre-MDT0002 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Autotest: Test running for 240 minutes (lustre-reviews_review-dne-part-3_113152.5) Lustre: DEBUG MARKER: debugfs -R 'stat /ROOT/panda' /dev/mapper/mds1_flakey | grep trusted.big Lustre: DEBUG MARKER: debugfs -w -R "ln <167> /lost+found" /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm11.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm12.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-\*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount FULL mdc.lustre-MDT0003-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm3.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: onyx-81vm3.onyx.whamcloud.com: executing set_default_debug -1 all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\|IDLE\) osc.lustre-OST0000-osc-[-0-9a-f]\*.ost_server_uuid Lustre: DEBUG MARKER: onyx-81vm2.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: onyx-81vm1.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace | Link to test |
| sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 Lustre: 123796:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3 previous similar messages PGD 0 P4D 0 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC CPU: 0 PID: 126982 Comm: lfsck Kdump: loaded Tainted: G W O -------- - - 4.18.0rh8.10-debug #7 Lustre: 123796:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.3-1.fc39 04/01/2014 RIP: 0010:osd_iit_iget+0x6ad/0x740 [osd_ldiskfs] Lustre: 123796:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3 previous similar messages Code: f5 fe 48 c7 c7 80 22 93 c1 48 83 05 a4 43 06 00 01 e8 e7 9d f5 fe 48 83 05 b7 4e 06 00 01 e9 4d fc ff ff 4c 8b 83 f8 db 00 00 <49> 3b 38 74 4b 48 8b 05 ff 4e 06 00 48 8d 48 01 31 c0 eb 11 48 63 RSP: 0018:ffffb85bd04dbc38 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff9ab050320000 RCX: 0000000000000000 RDX: 0000000000000000 RSI: 0000000000000003 RDI: 0000000200000003 RBP: ffff9ab060e23a98 R08: 0000000000000000 R09: 0000000000000000 Lustre: 123796:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/260/0 R10: ffffffffc17965e0 R11: ffffffffc1908789 R12: 0000000000004e5a R13: 0000000000000002 R14: 0000000000000000 R15: ffff9aaf5c309468 FS: 0000000000000000(0000) GS:ffff9ab081200000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000013e028001 CR4: 0000000000170ef0 Call Trace: ? show_regs.cold.9+0x22/0x2f ? __die_body+0x22/0x90 ? __die+0x33/0x4a Lustre: 123796:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3 previous similar messages ? no_context+0x30f/0x5a0 ? __bad_area_nosemaphore+0x1c6/0x260 ? bad_area_nosemaphore+0x1a/0x30 ? do_user_addr_fault+0x540/0x8a0 Lustre: 123796:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 1/12/0, punch: 0/0/0, quota 4/150/2 ? __do_page_fault+0x6b/0xa0 ? do_page_fault+0x87/0x4b6 ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x40/0x40 [ldiskfs] Lustre: 123796:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3 previous similar messages ? osd_iit_iget+0x6ad/0x740 [osd_ldiskfs] Lustre: 123796:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 1/17/2, delete: 0/0/0 ? osd_iit_iget+0x308/0x740 [osd_ldiskfs] Lustre: 123796:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3 previous similar messages osd_preload_next+0xa5/0xe0 [osd_ldiskfs] Lustre: 123796:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 osd_inode_iteration+0x463/0x1130 [osd_ldiskfs] Lustre: 123796:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3 previous similar messages ? osd_iit_iget+0x740/0x740 [osd_ldiskfs] ? osd_obj_spec_lookup+0x3c0/0x3c0 [osd_ldiskfs] osd_otable_it_next+0x223/0x8f0 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x3b3/0x1bb0 [lfsck] lfsck_master_engine+0x632/0x1300 [lfsck] ? woken_wake_function+0x30/0x30 ? lfsck_prep+0xd90/0xd90 [lfsck] kthread+0x1d7/0x210 ? set_kthread_struct+0x70/0x70 ret_from_fork+0x1f/0x30 Modules linked in: lustre(O) osp(O) ofd(O) lod(O) mdt(O) mdd(O) mgs(O) osd_ldiskfs(O) ldiskfs(O) lquota(O) lfsck(O) obdecho(O) mgc(O) mdc(O) lov(O) osc(O) lmv(O) fid(O) fld(O) ptlrpc_gss(O) ptlrpc(O) obdclass(O) ksocklnd(O) lnet(O) libcfs(O) dm_flakey zfs(O) spl(O) rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver intel_rapl_msr intel_rapl_common sb_edac rapl pcspkr i2c_piix4 squashfs crct10dif_pclmul crc32_pclmul ata_generic crc32c_intel ghash_clmulni_intel serio_raw ata_piix libata dm_mirror dm_region_hash dm_log dm_mod sha512_ssse3 sha512_generic [last unloaded: libcfs] CR2: 0000000000000000 | Lustre: 119775:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 259 < left 275, rollback = 2 Lustre: 119775:0:(osd_internal.h:1326:osd_trans_exec_op()) Skipped 3 previous similar messages Lustre: 119775:0:(osd_handler.c:1967:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 119775:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 119775:0:(osd_handler.c:1974:osd_trans_dump_creds()) attr_set: 1/1/1, xattr_set: 3/275/0 Lustre: 119775:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 119775:0:(osd_handler.c:1984:osd_trans_dump_creds()) write: 1/12/0, punch: 1/4/0, quota 1/3/0 Lustre: 119775:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 119775:0:(osd_handler.c:1991:osd_trans_dump_creds()) insert: 0/0/0, delete: 0/0/0 Lustre: 119775:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: 119775:0:(osd_handler.c:1998:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 Lustre: 119775:0:(osd_handler.c:1998:osd_trans_dump_creds()) Skipped 3 previous similar messages Lustre: *** cfs_fail_loc=1618, val=0*** Lustre: Skipped 3 previous similar messages LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 119416:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1742615682 with bad export cookie 6042259010626834130 LustreError: MGC192.168.206.110@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 119416:0:(ldlm_lockd.c:2591:ldlm_cancel_handler()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-OST0000 complete Lustre: 106002:0:(client.c:2346:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1742615682/real 1742615682] req@ffff9aaf536ad680 x1827264380861440/t0(0) o400->lustre-MDT0001-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1742615698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: oleg610-server.virtnet: executing load_modules_local LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 123796:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LustreError: 123795:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 123795:0:(ldlm_lib.c:1103:target_handle_connect()) Skipped 1 previous similar message LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: 124497:0:(mgc_request_server.c:550:mgc_llog_local_copy()) MGC192.168.206.110@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: Modifying parameter general.debug_raw_pointers in log params LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all LustreError: 125812:0:(ldlm_lib.c:1103:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:5 to 0x280000400:65) LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:4 to 0x2c0000400:65) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:5 to 0x2c0000401:65) Lustre: DEBUG MARKER: oleg610-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: Modifying parameter general.lod.*.mdt_hash in log params Lustre: 123796:0:(osd_internal.h:1326:osd_trans_exec_op()) lustre-OST0000: opcode 2: before 252 < left 260, rollback = 2 | Link to test |
| sanity-lfsck test 41: SEL support in LFSCK | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 467073 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.40.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffffb5e24a957c38 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff9a8e3ad68000 RCX: 0000000000000003 RDX: 0000000000000018 RSI: 0000000200000003 RDI: 0000000000000000 RBP: ffff9a8e223bd8c0 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc087f4f0 R11: ffff9a8d35981a10 R12: 0000000000004e57 R13: 0000000000000002 R14: 0000000000000000 R15: ffff9a8d35981788 FS: 0000000000000000(0000) GS:ffff9a8e3bd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000004c410005 CR4: 00000000003706e0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] ? osd_iit_iget+0x1f9/0x4f0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4f0/0x4f0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] ? __schedule+0x2d9/0x870 ? prepare_to_wait_event+0xa0/0x180 osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common dns_resolver crct10dif_pclmul nfs crc32_pclmul lockd grace fscache ghash_clmulni_intel joydev pcspkr i2c_piix4 virtio_balloon sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata virtio_net crc32c_intel virtio_blk net_failover serio_raw failover [last unloaded: dm_flakey] CR2: 0000000000000000 | Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+lfsck Lustre: DEBUG MARKER: /usr/sbin/lctl dk Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -A -t all -r -n on | Link to test |
| sanity-lfsck test 18e: Find out orphan OST-object and repair it (5) | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 408029 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.27.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffff9b5341c33c38 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8d8880cb0000 RCX: 0000000000000003 RDX: 0000000000000018 RSI: 0000000200000003 RDI: 0000000000000000 RBP: ffff8d8889f47540 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc09b3c40 R11: ffff8d887f23a3f5 R12: 0000000000004e48 R13: 0000000000000002 R14: 0000000000000000 R15: ffff8d8887733300 FS: 0000000000000000(0000) GS:ffff8d88ffd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 000000007c010002 CR4: 00000000001706e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] ? osd_iit_iget+0x1f9/0x4f0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4f0/0x4f0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss intel_rapl_msr nfsv4 intel_rapl_common dns_resolver crct10dif_pclmul nfs lockd grace fscache crc32_pclmul ghash_clmulni_intel pcspkr i2c_piix4 joydev virtio_balloon sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net net_failover serio_raw failover virtio_blk [last unloaded: dm_flakey] CR2: 0000000000000000 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1618 Lustre: *** cfs_fail_loc=1618, val=0*** Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; LustreError: 393104:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) ldlm_cancel from 10.240.28.114@tcp arrived at 1733951838 with bad export cookie 14982777335068016028 LustreError: 393104:0:(ldlm_lockd.c:2575:ldlm_cancel_handler()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds3' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds3 Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds3' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc LustreError: 402794:0:(ldlm_lib.c:1093:target_handle_connect()) lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 402794:0:(ldlm_lib.c:1093:target_handle_connect()) Skipped 71 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-102vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-102vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds3_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds3; mount -t lustre -o localrecov /dev/mapper/mds3_flakey /mnt/lustre-mds3 LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n health_check Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-102vm10.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null | grep -E ':[a-zA-Z]{3}[0-9]{4}' Lustre: DEBUG MARKER: e2label /dev/mapper/mds3_flakey 2>/dev/null Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-102vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-102vm11.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-51vm4.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Autotest: Test running for 105 minutes (lustre-reviews_review-dne-part-2_109586.4) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-52vm5.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-52vm5.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush Lustre: DEBUG MARKER: eval /usr/sbin/lctl get_param osp.lustre-OST*-MDT0000.prealloc_last_id | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=10 fail_loc=0x1602 Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n debug Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=-1 debug_mb=150 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o -c | Link to test |
| sanity-lfsck test 36a: rebuild LOV EA for mirrored file (1) | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 0 PID: 454693 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.16.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffff9d44883b7c38 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff9047420f8000 RCX: 0000000000000003 RDX: 0000000000000018 RSI: 0000000200000003 RDI: 0000000000000000 RBP: ffff90473e510a80 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc099a5d0 R11: ffff904727c8da10 R12: 0000000000004e50 R13: 0000000000000002 R14: 0000000000000000 R15: ffff904727c8d710 FS: 0000000000000000(0000) GS:ffff9047bfc00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000075c10004 CR4: 00000000003706f0 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x235/0x4f0 [osd_ldiskfs] ? osd_iit_iget+0x1f9/0x4f0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4f0/0x4f0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] ? __schedule+0x2d9/0x870 ? lfsck_layout_get_lovea.part.75+0x70/0x260 [lfsck] ? prepare_to_wait_event+0xa0/0x180 osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 intel_rapl_msr dns_resolver intel_rapl_common crct10dif_pclmul crc32_pclmul nfs lockd ghash_clmulni_intel grace fscache pcspkr joydev virtio_balloon i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix virtio_net libata crc32c_intel serio_raw net_failover failover virtio_blk [last unloaded: dm_flakey] CR2: 0000000000000000 | Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1616 Lustre: *** cfs_fail_loc=1616, val=0*** Lustre: *** cfs_fail_loc=1616, val=0*** Lustre: Skipped 34 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t layout -r -o | Link to test |
| sanity-scrub test 11: OI scrub skips the new created objects only once | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 310572 Comm: lfsck Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.8.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x233/0x4e0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffffa9fd0856fc38 EFLAGS: 00010246 RAX: 0000000000000002 RBX: ffff8b1ea5d18000 RCX: 0000000000000003 RDX: 0000000000000000 RSI: 0000000000000001 RDI: 0000000000000000 RBP: ffff8b1ed6136700 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc086a870 R11: ffff8b1ea7770a10 R12: 000000000000fa06 R13: 0000000000000002 R14: 0000000000000000 R15: ffff8b1ea77702d8 FS: 0000000000000000(0000) GS:ffff8b1f3fd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000097c10006 CR4: 00000000000606e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x157/0x180 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x233/0x4e0 [osd_ldiskfs] ? osd_iit_iget+0x1f7/0x4e0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4e0/0x4e0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] ? __schedule+0x2d9/0x870 ? prepare_to_wait_event+0xa0/0x180 osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr virtio_balloon i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata crc32c_intel virtio_net serio_raw net_failover virtio_blk failover [last unloaded: dm_flakey] CR2: 0000000000000000 | Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds2' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds4' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds2_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds2_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds2; mount -t lustre -o localrecov /dev/mapper/mds2_flakey /mnt/lustre-mds2 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds4_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds4_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds4; mount -t lustre -o localrecov /dev/mapper/mds4_flakey /mnt/lustre-mds4 Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-95vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: trevis-95vm5.trevis.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0001 -t scrub 3 -r Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0003 -t scrub 3 -r | Link to test |
| sanity-scrub test 11: OI scrub skips the new created objects only once | BUG: unable to handle kernel NULL pointer dereference at 0000000000000000 PGD 0 P4D 0 Oops: 0000 [#1] SMP PTI CPU: 1 PID: 130808 Comm: lfsck Kdump: loaded Tainted: G OE --------- - - 4.18.0-513.24.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 RIP: 0010:osd_iit_iget+0x233/0x4e0 [osd_ldiskfs] Code: 8b 37 48 83 fe 01 74 1b 48 83 fe 0a 74 15 48 b8 fd ff ff ff fd ff ff ff 48 01 f0 48 a9 fd ff ff ff 75 2e 48 8b bb 40 58 00 00 <48> 39 37 0f 84 8b 02 00 00 31 d2 eb 13 48 63 c2 48 c1 e0 04 48 01 RSP: 0018:ffffa8b202e43c38 EFLAGS: 00010246 RAX: 0000000000000002 RBX: ffff99da6c930000 RCX: 0000000000000004 RDX: 0000000000000000 RSI: 0000000200000005 RDI: 0000000000000000 RBP: ffff99da631a05c0 R08: 0000000000000000 R09: 0000000000000061 R10: ffffffffc0730760 R11: ffff99da6d3faa10 R12: 000000000001771d R13: 0000000000000002 R14: 0000000000000000 R15: ffff99da6d3fa008 FS: 0000000000000000(0000) GS:ffff99daffd00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 0000000000000000 CR3: 0000000075610002 CR4: 00000000000606e0 Call Trace: ? __die_body+0x1a/0x60 ? no_context+0x1ba/0x3f0 ? __bad_area_nosemaphore+0x16c/0x1c0 ? do_page_fault+0x37/0x12d ? page_fault+0x1e/0x30 ? ldiskfs_xattr_trusted_set+0x20/0x20 [ldiskfs] ? osd_iit_iget+0x233/0x4e0 [osd_ldiskfs] ? osd_iit_iget+0x1f7/0x4e0 [osd_ldiskfs] osd_preload_next+0x8e/0xa0 [osd_ldiskfs] osd_inode_iteration+0x3af/0xc20 [osd_ldiskfs] ? osd_iit_iget+0x4e0/0x4e0 [osd_ldiskfs] ? osd_obj_spec_lookup+0x290/0x290 [osd_ldiskfs] ? __schedule+0x2d9/0x870 ? prepare_to_wait_event+0xa0/0x180 osd_otable_it_next+0x1b1/0x630 [osd_ldiskfs] ? init_wait_var_entry+0x50/0x50 lfsck_master_oit_engine+0x2ac/0x1380 [lfsck] lfsck_master_engine+0x4c1/0xce0 [lfsck] ? finish_wait+0x80/0x80 ? lfsck_master_oit_engine+0x1380/0x1380 [lfsck] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 Modules linked in: dm_flakey osp(OE) mdd(OE) lod(OE) mdt(OE) lfsck(OE) mgs(OE) mgc(OE) osd_ldiskfs(OE) lquota(OE) lustre(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ksocklnd(OE) ptlrpc(OE) obdclass(OE) lnet(OE) ldiskfs(OE) libcfs(OE) dm_mod rpcsec_gss_krb5 auth_rpcgss nfsv4 dns_resolver nfs lockd grace fscache intel_rapl_msr intel_rapl_common crct10dif_pclmul crc32_pclmul ghash_clmulni_intel joydev pcspkr virtio_balloon i2c_piix4 sunrpc ext4 mbcache jbd2 ata_generic ata_piix libata virtio_net crc32c_intel net_failover serio_raw virtio_blk failover [last unloaded: dm_flakey] CR2: 0000000000000000 | Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre-mds1' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1 Lustre: DEBUG MARKER: modprobe dm-flakey; Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey >/dev/null 2>&1 Lustre: DEBUG MARKER: dmsetup status /dev/mapper/mds1_flakey 2>&1 Lustre: DEBUG MARKER: test -b /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: e2label /dev/mapper/mds1_flakey Lustre: DEBUG MARKER: mkdir -p /mnt/lustre-mds1; mount -t lustre -o localrecov /dev/mapper/mds1_flakey /mnt/lustre-mds1 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P debug_raw_pointers=Y Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-43vm2.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: onyx-43vm2.onyx.whamcloud.com: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all Lustre: DEBUG MARKER: lctl get_param -n timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Using TIMEOUT=20 Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: DEBUG MARKER: [ -f /sys/module/mgc/parameters/mgc_requeue_timeout_min ] && echo 1 > /sys/module/mgc/parameters/mgc_requeue_timeout_min; exit 0 Lustre: DEBUG MARKER: lctl dl | grep ' IN osc ' 2>/dev/null | wc -l Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-ldiskfs.lustre-MDT0000.quota_slave.enabled Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.mdt=ug3 Lustre: DEBUG MARKER: /usr/sbin/lctl conf_param lustre.quota.ost=ug3 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -P lod.*.mdt_hash=crush Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 3 -r Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub | Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n osd-*.lustre-MDT0000.oi_scrub Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t scrub 6 -r | Link to test |