| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| conf-sanity test 44: mounted client proc entry exists | BUG: unable to handle kernel paging request at ffff88028dc202f8 IP: [<ffffffffa13a0245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb8b067 PTE 800000028dc20060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm serio_raw drm_panel_orientation_quirks libata i2c_core floppy virtio_blk [last unloaded: libcfs] CPU: 6 PID: 6717 Comm: kworker/u32:5 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88027aa80010 ti: ffff8802e18a0000 task.ti: ffff8802e18a0000 RIP: 0010:[<ffffffffa13a0245>] [<ffffffffa13a0245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8802e18a3cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802e88b9158 RCX: 0000000000000000 RDX: 0000000000000006 RSI: ffff88028dc202f0 RDI: ffff88026ba89ef8 RBP: ffff8802e18a3ce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88028dc20000 R13: ffff8802e18a3cf0 R14: ffffffffa14062c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88028dc202f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0385694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0385ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03876f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa039cb0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa039d298>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0381138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03815c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.62_96_g1f97e88 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:131 to 0x240000402:161) Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 30752:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1714327148 with bad export cookie 16743677886945202316 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| conf-sanity test 21a: start mds before ost, stop ost first | BUG: unable to handle kernel paging request at ffff8802deb50160 IP: [<ffffffffa0f66245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e903067 PTE 80000002deb50060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_ldiskfs(OE) ldiskfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) dm_flakey dm_mod loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) jbd2 mbcache crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy i2c_core libata serio_raw virtio_blk [last unloaded: libcfs] CPU: 14 PID: 12136 Comm: kworker/u32:1 Kdump: loaded Tainted: P B OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802d2d70010 ti: ffff88025f798000 task.ti: ffff88025f798000 RIP: 0010:[<ffffffffa0f66245>] [<ffffffffa0f66245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff88025f79bcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880271bf33f8 RCX: 0000000000000000 RDX: 000000000000000e RSI: ffff8802deb50158 RDI: ffff880080433ee8 RBP: ffff88025f79bce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802deb50000 R13: ffff88025f79bcf0 R14: ffffffffa0fcc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802deb50160 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03aa694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03aaad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03ac6f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c1b0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c2298>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a6138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a65c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 4 previous similar messages LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LDISKFS-fs (dm-3): file extents enabled, maximum tree depth=5 LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: user_xattr,acl,no_mbcache,nodelalloc Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:36 to 0x2c0000402:65) Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 9 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 14 previous similar messages LustreError: 18131:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1714029022 with bad export cookie 12304721217008372359 LustreError: 18131:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.36@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete | Link to test |
| conf-sanity test 21e: separate MGS and MDS | BUG: unable to handle kernel paging request at ffff8802fc5f82f8 IP: [<ffffffffa1386245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e816067 PTE 80000002fc5f8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core floppy libata [last unloaded: libcfs] CPU: 5 PID: 404 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031b1ba4f0 ti: ffff88031957c000 task.ti: ffff88031957c000 RIP: 0010:[<ffffffffa1386245>] [<ffffffffa1386245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff88031957fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88008d748008 RCX: 0000000000000000 RDX: 0000000000000005 RSI: ffff8802fc5f82f0 RDI: ffff88009b635658 RBP: ffff88031957fce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802fc5f8000 R13: ffff88031957fcf0 R14: ffffffffa13ec2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802fc5f82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0394694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0394ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03966f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03abb0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ac298>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0390138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03905c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: 25560:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.11@tcp: no remote llog for test1234-sptlrpc, check MGS config Lustre: 25560:0:(mgc_request_server.c:597:mgc_process_server_cfg_log()) MGC192.168.123.11@tcp: local log test1234-sptlrpc are not valid and/or remote logs are not accessbile rc = -2 Lustre: test1234-OST0000: new disk, initializing Lustre: srv-test1234-OST0000: No data found on store. Initialize space. Lustre: Setting parameter test1234-MDT0000.mdt.identity_upcall in log test1234-MDT0000 Lustre: ctl-test1234-MDT0000: No data found on store. Initialize space. Lustre: test1234-MDT0000: new disk, initializing Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-test1234-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 27298:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff88031da22340 x1797251039009472/t0(0) o700->test1234-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1713992579 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted test1234-client Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-test1234-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: test1234-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 Lustre: Unmounted test1234-client Lustre: test1234-MDT0000-lwp-OST0000: Connection to test1234-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: test1234-MDT0000: Not available for connect from 0@lo (stopping) | Link to test |
| conf-sanity test 21e: separate MGS and MDS | BUG: unable to handle kernel paging request at ffff88023d9982f8 IP: [<ffffffffa1384245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33effc067 PMD 33ee0f067 PTE 800000023d998060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk floppy libata i2c_core serio_raw [last unloaded: libcfs] CPU: 7 PID: 31423 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802d00bc9d0 ti: ffff88027cb3c000 task.ti: ffff88027cb3c000 RIP: 0010:[<ffffffffa1384245>] [<ffffffffa1384245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff88027cb3fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802ab2ec548 RCX: 0000000000000000 RDX: 0000000000000007 RSI: ffff88023d9982f0 RDI: ffff8800b4febb68 RBP: ffff88027cb3fce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88023d998000 R13: ffff88027cb3fcf0 R14: ffffffffa13ea2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88023d9982f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa039d694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa039dad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa039f6f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b4b0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b5708>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0399138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03995c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: 32459:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.76@tcp: no remote llog for test1234-sptlrpc, check MGS config Lustre: 32459:0:(mgc_request_server.c:597:mgc_process_server_cfg_log()) MGC192.168.123.76@tcp: local log test1234-sptlrpc are not valid and/or remote logs are not accessbile rc = -2 Lustre: test1234-OST0000: new disk, initializing Lustre: srv-test1234-OST0000: No data found on store. Initialize space. Lustre: Setting parameter test1234-MDT0000.mdt.identity_upcall in log test1234-MDT0000 Lustre: ctl-test1234-MDT0000: No data found on store. Initialize space. Lustre: test1234-MDT0000: new disk, initializing Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-test1234-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 1789:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff880277061440 x1797137384902912/t0(0) o700->test1234-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1713884199 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted test1234-client Lustre: ctl-test1234-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-test1234-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: test1234-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 Lustre: Unmounted test1234-client Lustre: test1234-MDT0000-lwp-OST0000: Connection to test1234-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: test1234-MDT0000: Not available for connect from 0@lo (stopping) | Link to test |
| conf-sanity test 50a: lazystatfs all servers available | BUG: unable to handle kernel paging request at ffff8802b26182f8 IP: [<ffffffffa1384245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea66067 PTE 80000002b2618060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks floppy ata_piix virtio_blk serio_raw i2c_core libata [last unloaded: libcfs] CPU: 1 PID: 497 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c202dc40 ti: ffff8802e2458000 task.ti: ffff8802e2458000 RIP: 0010:[<ffffffffa1384245>] [<ffffffffa1384245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8802e245bcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802ec7bd698 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffff8802b26182f0 RDI: ffff880258ca25c8 RBP: ffff8802e245bce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802b2618000 R13: ffff8802e245bcf0 R14: ffffffffa13ea2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802b26182f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa038c694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa038cad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa038e6f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a3b0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03a4708>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0388138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03885c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.62_64_gfed8e33 LNet: Added LNI 192.168.123.106@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3 to 0x2c0000402:33) Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 30916:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713533033 with bad export cookie 3359709199116762533 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff8802fa0502f8 IP: [<ffffffffa1396245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e828067 PTE 80000002fa050060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs] CPU: 11 PID: 2397 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800a92224f0 ti: ffff880232118000 task.ti: ffff880232118000 RIP: 0010:[<ffffffffa1396245>] [<ffffffffa1396245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff88023211bcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88008a9233f8 RCX: 0000000000000000 RDX: 000000000000000b RSI: ffff8802fa0502f0 RDI: ffff8802b5fd46b8 RBP: ffff88023211bce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802fa050000 R13: ffff88023211bcf0 R14: ffffffffa13fc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802fa0502f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03e5694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03e5ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03e76f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03fcb0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03fd708>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03e1138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03e15c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:35 to 0x240000400:97) Lustre: Mounted lustre-client Lustre: 27929:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.56@tcp: no remote llog for fs15246-sptlrpc, check MGS config Lustre: 27929:0:(mgc_request_server.c:597:mgc_process_server_cfg_log()) MGC192.168.123.56@tcp: local log fs15246-sptlrpc are not valid and/or remote logs are not accessbile rc = -2 Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 29508:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff880236f50f40 x1796660058408704/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1713428684 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client LustreError: 29508:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 29508:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 29508:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) | Link to test |
| conf-sanity test 42: allow client/server mount/unmount with invalid config param | BUG: unable to handle kernel paging request at ffff8800af5b02f8 IP: [<ffffffffa138c245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f886067 PTE 80000000af5b0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy libata serio_raw i2c_core virtio_blk [last unloaded: libcfs] CPU: 2 PID: 13798 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8803275d5c40 ti: ffff8802f9260000 task.ti: ffff8802f9260000 RIP: 0010:[<ffffffffa138c245>] [<ffffffffa138c245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8802f9263cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88008d9aa2a8 RCX: 0000000000000000 RDX: 0000000000000002 RSI: ffff8800af5b02f0 RDI: ffff8802d225f0c8 RBP: ffff8802f9263ce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8800af5b0000 R13: ffff8802f9263cf0 R14: ffffffffa13f22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800af5b02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0376694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0376ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03786f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa038db0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa038e708>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0372138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03725c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.62_64_gfed8e33 LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000402:67 to 0x240000402:97) Lustre: Mounted lustre-client Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 18232:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713365018 with bad export cookie 14622894059605734172 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages | BUG: unable to handle kernel paging request at ffff88027fff02f8 IP: [<ffffffffa1379078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ebfb067 PTE 800000027fff0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy i2c_core serio_raw virtio_blk libata [last unloaded: libcfs] CPU: 6 PID: 3208 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802a9eaa4f0 ti: ffff8802591e0000 task.ti: ffff8802591e0000 RIP: 0010:[<ffffffffa1379078>] [<ffffffffa1379078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802591e3cc8 EFLAGS: 00010246 RAX: ffffffffa13d0800 RBX: ffff88028517d698 RCX: 0000000000000000 RDX: 0000000000000001 RSI: ffff88027fff02f0 RDI: ffff8802591e3cf0 RBP: ffff8802591e3ce0 R08: 000000000000ffff R09: 000000000000ffff R10: 0000000000000000 R11: 000000000000000f R12: ffff88027fff0000 R13: ffff8802591e3cf0 R14: ffffffffa13df2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88027fff02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a4694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a4ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a66f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03bbb0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03bc708>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a0138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a05c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.62_64_gfed8e33 LNet: Added LNI 192.168.123.106@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:41 to 0x2c0000401:129) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000bd0:35 to 0x2c0000bd0:65) Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000400:35 to 0x2c0000400:129) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 7765:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713353092 with bad export cookie 13881644216202115910 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 7765:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message | Link to test |
| conf-sanity test 46a: handle ost additional - wide striped file | BUG: unable to handle kernel paging request at ffff8802f1f602f8 IP: [<ffffffffa13a4245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e869067 PTE 80000002f1f60060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs] CPU: 0 PID: 28911 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c1be49d0 ti: ffff8802cd374000 task.ti: ffff8802cd374000 RIP: 0010:[<ffffffffa13a4245>] [<ffffffffa13a4245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8802cd377cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802c1fdb3f8 RCX: 0000000000000000 RDX: 0000000000000000 RSI: ffff8802f1f602f0 RDI: ffff8800aea8ae98 RBP: ffff8802cd377ce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802f1f60000 R13: ffff8802cd377cf0 R14: ffffffffa140a2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802f1f602f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa037f694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa037fad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03816f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa0396b0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa0397708>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa037b138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa037b5c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-44.localnet: executing set_hostid alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.62_64_gfed8e33 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space. Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 24230:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1713307552 with bad export cookie 7870729174072913713 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 24230:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 3 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff8802897602f8 IP: [<ffffffffa1374078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ebae067 PTE 8000000289760060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk floppy serio_raw i2c_core libata [last unloaded: libcfs] CPU: 9 PID: 11380 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802f891dc40 ti: ffff8802cf8b4000 task.ti: ffff8802cf8b4000 RIP: 0010:[<ffffffffa1374078>] [<ffffffffa1374078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802cf8b7cc8 EFLAGS: 00010246 RAX: ffffffffa13cb800 RBX: ffff8800a90b5698 RCX: 0000000001000000 RDX: 0000000000000001 RSI: ffff8802897602f0 RDI: ffff8802cf8b7cf0 RBP: ffff8802cf8b7ce0 R08: ffff8802d5b85a58 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000009 R12: ffff880289760000 R13: ffff8802cf8b7cf0 R14: ffffffffa13da2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802897602f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0381694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0381ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03836f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa0398afe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03996f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa037d138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa037d5c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=19489 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-mdc-ffff8800a6041bf8: operation mds_sync to node 0@lo failed: rc = -19 LustreError: Skipped 21 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8800a6041bf8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 58 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 213 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages LustreError: 27482:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880070549940 x1796169079708864/t317827580331(317827580331) o101->lustre-MDT0000-mdc-ffff8800a6041bf8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712963640 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 27482:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 55 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3722 to 0x340000402:3745) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3722 to 0x300000402:3745) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3796 to 0x2c0000402:3841) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3722 to 0x380000402:3745) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:225) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:205 to 0x2c0000401:257) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:225) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:225) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0002-mdc-ffff8800a6041bf8: can't stat MDS #0: rc = -107 LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0002-mdc-ffff8800a6041bf8: can't stat MDS #0: rc = -19 LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0002-mdc-ffff8800a6041bf8: can't stat MDS #0: rc = -19 LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0002-mdc-ffff8800a6041bf8: can't stat MDS #0: rc = -19 LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) Skipped 1 previous similar message LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0002-mdc-ffff8800a6041bf8: can't stat MDS #0: rc = -19 LustreError: 19664:0:(lmv_obd.c:1451:lmv_statfs()) Skipped 2 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:3 to 0x380000400:33) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 166-1: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages LustreError: 27482:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880070549940 x1796169079708864/t317827580331(317827580331) o101->lustre-MDT0000-mdc-ffff8800a6041bf8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712963729 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 27482:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 21 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3849 to 0x340000402:3873) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3948 to 0x2c0000402:3969) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3853 to 0x300000402:3873) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3853 to 0x380000402:3873) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:257) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:257) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:257) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:205 to 0x2c0000401:289) Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 | Link to test |
| conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount | BUG: unable to handle kernel paging request at ffff8802938782f8 IP: [<ffffffffa1377245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb5d067 PTE 8000000293878060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks floppy ata_piix serio_raw virtio_blk i2c_core libata [last unloaded: libcfs] CPU: 4 PID: 15161 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880325bbb760 ti: ffff8802f7dac000 task.ti: ffff8802f7dac000 RIP: 0010:[<ffffffffa1377245>] [<ffffffffa1377245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8802f7dafcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802767933f8 RCX: 0000000000000000 RDX: 0000000000000004 RSI: ffff8802938782f0 RDI: ffff8802f0328b58 RBP: ffff8802f7dafce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880293878000 R13: ffff8802f7dafcf0 R14: ffffffffa13dd2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802938782f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b6694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b6ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b86f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03cdafe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ce6f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b2138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b25c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-94.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: 26784:0:(mgc_request_server.c:553:mgc_llog_local_copy()) MGC192.168.123.96@tcp: no remote llog for lustre-sptlrpc, check MGS config Lustre: 26784:0:(mgc_request_server.c:597:mgc_process_server_cfg_log()) MGC192.168.123.96@tcp: local log lustre-sptlrpc are not valid and/or remote logs are not accessbile rc = -2 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 27394:0:(lod_sub_object.c:966:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 LustreError: 27394:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space. Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: 28334:0:(lod_sub_object.c:966:lod_sub_prep_llog()) lustre-MDT0002-mdtlov: can't create new llog: rc = -115 LustreError: 28334:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0002-osd: get update log duration 0, retries 0, failed: rc = -115 Lustre: 28335:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802f99c3c40 x1796151102698496/t0(0) o700->lustre-MDT0001-osp-MDT0002@0@lo:30/10 lens 264/248 e 0 to 0 dl 1712943309 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: srv-lustre-MDT0002: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 28729:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802c598a840 x1796151102699968/t0(0) o700->lustre-MDT0002-osp-MDT0001@0@lo:30/10 lens 264/248 e 0 to 0 dl 1712943312 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space. Lustre: DEBUG MARKER: centos-94.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 166-1: MGC192.168.123.96@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: MGS: Client 344ef8a0-36d3-46c5-a026-7d6bf95457ea (at 0@lo) reconnecting Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 LustreError: 26383:0:(mgs_handler.c:264:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000402 Lustre: Mounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 31047:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8802bdc38008: inode [0x200000403:0x1:0x0] mdc close failed: rc = -108 Lustre: 15521:0:(llite_lib.c:4078:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.96@tcp:/lustre/fid: [0x200000403:0x1:0x0]/ may get corrupted (rc -108) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88028b9182f8 IP: [<ffffffffa138f078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb9d067 PTE 800000028b918060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw virtio_blk i2c_core floppy [last unloaded: libcfs] CPU: 15 PID: 26364 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008f00a4f0 ti: ffff8802c4bc0000 task.ti: ffff8802c4bc0000 RIP: 0010:[<ffffffffa138f078>] [<ffffffffa138f078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802c4bc3cc8 EFLAGS: 00010246 RAX: ffffffffa13e6800 RBX: ffff8800a3fc0008 RCX: 0000000001000000 RDX: 0000000000000001 RSI: ffff88028b9182f0 RDI: ffff8802c4bc3cf0 RBP: ffff8802c4bc3ce0 R08: ffff8802c51d5a58 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000f R12: ffff88028b918000 R13: ffff8802c4bc3cf0 R14: ffffffffa13f52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88028b9182f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c4694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c4ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c66f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03dbafe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03dc6f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03c0138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c05c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=24231 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages LustreError: 30813:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802d8c4b740 x1796130214362368/t317827580331(317827580331) o101->lustre-MDT0000-mdc-ffff8802cbbe4138@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712926484 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 30813:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3788 to 0x340000402:3809) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3820 to 0x300000402:3841) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3788 to 0x380000402:3809) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3766 to 0x2c0000402:3809) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-mdc-ffff8802cbbe4138: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 13 previous similar messages Lustre: lustre-MDT0001-mdc-ffff8802cbbe4138: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 66 previous similar messages LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 158 previous similar messages Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 63 previous similar messages Lustre: lustre-MDT0001: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:225) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:289) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:289) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:289) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:3 to 0x380000400:33) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:3 to 0x340000400:33) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 24482:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0000-mdc-ffff8802cbbe4138: can't stat MDS #0: rc = -107 LustreError: 30813:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802d8c4b740 x1796130214362368/t317827580331(317827580331) o101->lustre-MDT0000-mdc-ffff8802cbbe4138@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712926561 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 30813:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 29 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3922 to 0x380000402:3937) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3911 to 0x2c0000402:3937) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3913 to 0x340000402:3937) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3948 to 0x300000402:3969) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:202 to 0x380000401:321) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:204 to 0x2c0000401:321) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:203 to 0x300000401:257) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:321) Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 | Link to test |
| replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 | BUG: unable to handle kernel paging request at ffff88025f4982f8 IP: [<ffffffffa137f078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ed00067 PTE 800000025f498060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs] CPU: 8 PID: 13019 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880085c2c9d0 ti: ffff88026501c000 task.ti: ffff88026501c000 RIP: 0010:[<ffffffffa137f078>] [<ffffffffa137f078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88026501fcc8 EFLAGS: 00010246 RAX: ffffffffa13d6800 RBX: ffff8802ea138008 RCX: 0000000001000000 RDX: 0000000000000001 RSI: ffff88025f4982f0 RDI: ffff88026501fcf0 RBP: ffff88026501fce0 R08: ffff88028a5fcfb8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000008 R12: ffff88025f498000 R13: ffff88026501fcf0 R14: ffffffffa13e52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88025f4982f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0392694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0392ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03946f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a9afe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03aa6f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa038e138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa038e5c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 LustreError: 166-1: MGC192.168.123.43@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 6 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 40 previous similar messages | Link to test |
| conf-sanity test 50f: normal statfs one server in down | BUG: unable to handle kernel paging request at ffff8802ba5582f8 IP: [<ffffffffa136c245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea27067 PTE 80000002ba558060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm ata_piix drm_panel_orientation_quirks virtio_blk floppy i2c_core serio_raw libata [last unloaded: libcfs] CPU: 10 PID: 12276 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800922e8010 ti: ffff8800ae7c8000 task.ti: ffff8800ae7c8000 RIP: 0010:[<ffffffffa136c245>] [<ffffffffa136c245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8800ae7cbcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88009fbd1158 RCX: 0000000000000000 RDX: 000000000000000a RSI: ffff8802ba5582f0 RDI: ffff8802be24b398 RBP: ffff8800ae7cbce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802ba558000 R13: ffff8800ae7cbcf0 R14: ffffffffa13d22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802ba5582f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a1694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a1ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a36f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b8afe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b96f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039d138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039d5c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 9 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0001: new disk, initializing Lustre: Skipped 3 previous similar messages Lustre: srv-lustre-OST0001: No data found on store. Initialize space. Lustre: Skipped 4 previous similar messages Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:ost Lustre: Skipped 3 previous similar messages Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost] Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in DISCONN state after 0 sec Lustre: Mounted lustre-client LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 27 previous similar messages Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid LustreError: 167-0: lustre-OST0001-osc-MDT0002: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: lustre-OST0001-osc-MDT0002: Connection restored to (at 0@lo) Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 28 previous similar messages Lustre: server umount lustre-OST0001 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 15685:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712867943 with bad export cookie 4003635739661853309 LustreError: 15685:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88027aa382f8 IP: [<ffffffffa138f078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ec25067 PTE 800000027aa38060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy virtio_blk libata i2c_core [last unloaded: libcfs] CPU: 0 PID: 28700 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800849349d0 ti: ffff88024bd98000 task.ti: ffff88024bd98000 RIP: 0010:[<ffffffffa138f078>] [<ffffffffa138f078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88024bd9bcc8 EFLAGS: 00010246 RAX: ffffffffa13e6800 RBX: ffff88025010b3f8 RCX: 0000000001000000 RDX: 0000000000000001 RSI: ffff88027aa382f0 RDI: ffff88024bd9bcf0 RBP: ffff88024bd9bce0 R08: ffff8802c2a23fc8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000000 R12: ffff88027aa38000 R13: ffff88024bd9bcf0 R14: ffffffffa13f52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88027aa382f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa03b3694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b3ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b56f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03caafe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cb6f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03af138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03af5c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=12904 ... LustreError: 14161:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 14161:0:(osd_handler.c:698:osd_ro()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation out_update to node 0@lo failed: rc = -107 LustreError: Skipped 16 previous similar messages Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 58 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 222 previous similar messages LustreError: 166-1: MGC192.168.123.108@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 10 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-OST0003: Connection restored to 192.168.123.108@tcp (at 0@lo) Lustre: Skipped 56 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3745) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3706 to 0x300000402:3745) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3745) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3749 to 0x2c0000402:3777) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -107 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -19 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -19 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -19 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -19 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) Skipped 2 previous similar messages LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88009f747340 x1796045485411968/t4294969757(4294969757) o101->lustre-MDT0001-mdc-ffff88032581a548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712845886 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:289) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:289) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:205 to 0x2c0000401:257) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:225) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802f2a91440 x1796045485417408/t4294967346(4294967346) o101->lustre-MDT0002-mdc-ffff88032581a548@0@lo:12/10 lens 624/608 e 0 to 0 dl 1712845911 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 Lustre: lustre-OST0001: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x300000400:133 to 0x300000400:161) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:134 to 0x380000400:161) Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000400:133 to 0x2c0000400:161) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:134 to 0x340000400:161) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3706 to 0x300000402:3777) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3777) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3777) Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3749 to 0x2c0000402:3809) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -19 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) Skipped 3 previous similar messages LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88009f747340 x1796045485411968/t4294969757(4294969757) o101->lustre-MDT0001-mdc-ffff88032581a548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712845961 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 49 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:257) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:205 to 0x2c0000401:289) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:321) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:321) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000400:318 to 0x2c0000400:353) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:318 to 0x380000400:353) Lustre: lustre-OST0001: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x300000400:317 to 0x300000400:353) Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:317 to 0x340000400:353) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3749 to 0x2c0000402:3841) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3809) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3706 to 0x300000402:3809) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3809) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -107 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) Skipped 7 previous similar messages LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88009f747340 x1796045485411968/t4294969757(4294969757) o101->lustre-MDT0001-mdc-ffff88032581a548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712846037 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 52 previous similar messages Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:205 to 0x2c0000401:321) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:353) Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:289) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:353) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0002: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x340000400:517 to 0x340000400:577) Lustre: lustre-OST0001: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x300000400:517 to 0x300000400:577) Lustre: lustre-OST0000: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x2c0000400:516 to 0x2c0000400:577) Lustre: lustre-OST0003: new connection from lustre-MDT0002-mdtlov (cleaning up unused objects from 0x380000400:517 to 0x380000400:577) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:3749 to 0x2c0000402:3873) Lustre: lustre-OST0002: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x340000402:3706 to 0x340000402:3841) Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x300000402:3706 to 0x300000402:3841) Lustre: lustre-OST0003: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x380000402:3706 to 0x380000402:3841) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 11 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) lustre-MDT0001-mdc-ffff88032581a548: can't stat MDS #0: rc = -19 LustreError: 13146:0:(lmv_obd.c:1451:lmv_statfs()) Skipped 7 previous similar messages LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88009f747340 x1796045485411968/t4294969757(4294969757) o101->lustre-MDT0001-mdc-ffff88032581a548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1712846114 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 23185:0:(client.c:3294:ptlrpc_replay_interpret()) Skipped 25 previous similar messages Lustre: lustre-OST0003: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x380000401:203 to 0x380000401:321) Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x300000401:202 to 0x300000401:385) Lustre: lustre-OST0002: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x340000401:202 to 0x340000401:385) Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000401:205 to 0x2c0000401:353) Lustre: DEBUG MARKER: centos-106.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 12 times Lustre: Failing over lustre-MDT0002 | Link to test |
| conf-sanity test 23b: Simulate -EINTR during mount | BUG: unable to handle kernel paging request at ffff8802be9902f8 IP: [<ffffffffa1387245>] lod_device_free+0x215/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea05067 PTE 80000002be990060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy i2c_core virtio_blk libata [last unloaded: libcfs] CPU: 15 PID: 26222 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802f1f7c9d0 ti: ffff8802e4d8c000 task.ti: ffff8802e4d8c000 RIP: 0010:[<ffffffffa1387245>] [<ffffffffa1387245>] lod_device_free+0x215/0x2c0 [lod] RSP: 0018:ffff8802e4d8fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802ce68d698 RCX: 0000000000000000 RDX: 000000000000000f RSI: ffff8802be9902f0 RDI: ffff8800ad998788 RBP: ffff8802e4d8fce0 R08: 000000000000ffff R09: 000000000000ffff R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802be990000 R13: ffff8802e4d8fcf0 R14: ffffffffa13ed2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802be9902f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0381694>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0381ad0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03836f5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa0398afe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03996f8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa037d138>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa037d5c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.62_24_g791aab7 LNet: Added LNI 192.168.123.116@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000402:35 to 0x2c0000402:97) Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 17867:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712827991 with bad export cookie 12236066000214067044 LustreError: 166-1: MGC192.168.123.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one | BUG: unable to handle kernel paging request at ffff8802904482f8 IP: [<ffffffffa137c23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb77067 PTE 8000000290448060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw libata virtio_blk i2c_core floppy [last unloaded: libcfs] CPU: 2 PID: 32014 Comm: kworker/u32:5 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c4ed49d0 ti: ffff88022c854000 task.ti: ffff88022c854000 RIP: 0010:[<ffffffffa137c23d>] [<ffffffffa137c23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88022c857cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880243cc8008 RCX: ffff880289520d60 RDX: 0000000000000002 RSI: ffff8802904482f0 RDI: ffff8802923cc238 RBP: ffff88022c857ce0 R08: ffff880289520d70 R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880290448000 R13: ffff88022c857cf0 R14: ffffffffa13e22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802904482f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa039f5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa039f9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a1605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b6bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b77e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039b068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039b4f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Key type lgssc unregistered LNet: 14215:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.11@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.11@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3 to 0x240000401:33 Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 15243:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712791505 with bad export cookie 2404399954832173979 LustreError: 166-1: MGC192.168.123.11@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 19159:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.11@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.11@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 20255:0:(tgt_mount.c:2178:server_fill_super()) cfs_race id 716 sleeping LustreError: 20254:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 waking LustreError: 20255:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 awake: rc=4952 LustreError: 20255:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on lustre-mdt1/mdt1: -114 LustreError: 20255:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -114 LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-9.localnet: executing lsmod LustreError: 22102:0:(tgt_mount.c:2178:server_fill_super()) cfs_race id 716 sleeping LustreError: 22103:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 waking LustreError: 22102:0:(tgt_mount.c:2178:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 LustreError: 22102:0:(tgt_mount.c:2190:server_fill_super()) Unable to start osd on lustre-ost1/ost1: -114 LustreError: 22102:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -114 LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0000 complete LustreError: 20285:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712791559 with bad export cookie 7737455443434716709 LustreError: 166-1: MGC192.168.123.11@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 20908:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880098bac640 x1795992088229504/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 21478:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802d01c5a40 x1795992088230976/t0(0) o900->lustre-MDT0000-lwp-MDT0002@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0002 complete Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-9.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000401:3 to 0x240000401:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages LustreError: 23531:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712791618 with bad export cookie 7737455443434717745 LustreError: 23531:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.11@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: 19393:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1712791607/real 1712791607] req@ffff8802bf748a40 x1795992088251008/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1712791623 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 | Link to test |
| conf-sanity test 24a: Multiple MDTs on a single node | BUG: unable to handle kernel paging request at ffff880098a182f8 IP: [<ffffffffa137523d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f93b067 PTE 8000000098a18060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk libata i2c_core floppy [last unloaded: libcfs] CPU: 6 PID: 1224 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031d8049d0 ti: ffff8800a864c000 task.ti: ffff8800a864c000 RIP: 0010:[<ffffffffa137523d>] [<ffffffffa137523d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8800a864fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88029c709158 RCX: ffff8802b2de174e RDX: 0000000000000006 RSI: ffff880098a182f0 RDI: ffff88009d75c438 RBP: ffff8800a864fce0 R08: ffff8802b2de175e R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880098a18000 R13: ffff8800a864fcf0 R14: ffffffffa13db2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880098a182f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa037d5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa037d9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa037f605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa0394bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03957e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0379068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03794f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: Setting parameter 969362ae-MDT0000.mdt.identity_upcall in log 969362ae-MDT0000 Lustre: ctl-969362ae-MDT0000: No data found on store. Initialize space. Lustre: 969362ae-MDT0000: new disk, initializing Lustre: 969362ae-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-969362ae-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: 969362ae-OST0000: new disk, initializing Lustre: srv-969362ae-OST0000: No data found on store. Initialize space. Lustre: Skipped 1 previous similar message Lustre: ctl-969362ae-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-969362ae-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: 969362ae-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 Lustre: Mounted 969362ae-client Lustre: 969362ae-MDD0000: changelog on Lustre: 969362ae-MDD0000: changelog off Lustre: Failing over 969362ae-MDT0000 Lustre: server umount 969362ae-MDT0000 complete Lustre: 969362ae-MDT0000-mdc-ffff880280c15d28: Connection to 969362ae-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: 969362ae-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: 969362ae-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: 969362ae-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: 969362ae-MDT0000: in recovery but waiting for the first client to connect Lustre: Failing over 969362ae-OST0000 Lustre: server umount 969362ae-OST0000 complete Lustre: 969362ae-OST0000-osc-ffff880280c15d28: Connection to 969362ae-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: 969362ae-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: 969362ae-MDT0000: Will be in recovery for at least 1:00, or until 1 client reconnects Lustre: 969362ae-MDT0000: Recovery over after 0:01, of 1 clients 1 recovered and 0 were evicted. Lustre: 969362ae-MDT0000-mdc-ffff880280c15d28: Connection restored to 192.168.123.6@tcp (at 0@lo) LustreError: 137-5: 969362ae-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: 969362ae-OST0000: in recovery but waiting for the first client to connect Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: 969362ae-OST0000: Will be in recovery for at least 1:00, or until 2 clients reconnect Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 969362ae-OST0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. Lustre: 969362ae-OST0000-osc-MDT0000: Connection restored to 192.168.123.6@tcp (at 0@lo) Lustre: 969362ae-OST0000: deleting orphan objects from 0x240000400:3 to 0x240000400:33 Lustre: server umount lustre-MDT0000 complete LustreError: 25996:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712718986 with bad export cookie 14707979095080894621 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: 26014:0:(mgc_request.c:1844:mgc_process_log()) MGC192.168.123.6@tcp: IR log 969362ae-mdtir failed, not fatal: rc = -5 LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete | Link to test |
| conf-sanity test 101a: Race MDT->OST reconnection with create | BUG: unable to handle kernel paging request at ffff8802c1aa82f8 IP: [<ffffffffa139a23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e9eb067 PTE 80000002c1aa8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs] CPU: 0 PID: 11920 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031bacdc40 ti: ffff8802b2d70000 task.ti: ffff8802b2d70000 RIP: 0010:[<ffffffffa139a23d>] [<ffffffffa139a23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802b2d73cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88029266a2a8 RCX: ffff8802bde8103a RDX: 0000000000000000 RSI: ffff8802c1aa82f0 RDI: ffff8802f45137e8 RBP: ffff8802b2d73ce0 R08: ffff8802bde8104a R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802c1aa8000 R13: ffff8802b2d73cf0 R14: ffffffffa14002c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802c1aa82f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa03a05a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a09e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a2605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b7bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b87e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039c068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039c4f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.41@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:3 to 0x2c0000402:33 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 6 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 6 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 6 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 11 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 11 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 11 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 16 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 16 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 16 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 16 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 32 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 32 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 32 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 32 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 32 previous similar messages LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 87 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 87 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 87 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 87 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 87 previous similar messages LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 153 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 153 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 153 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 153 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 153 previous similar messages LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 272 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 273 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 273 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 273 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.41@tcp (at 0@lo) Lustre: Skipped 273 previous similar messages LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) Skipped 1 previous similar message LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) Skipped 1 previous similar message LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 2801:0:(osp_precreate.c:670:osp_precreate_send()) Skipped 2 previous similar messages LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 2801:0:(osp_precreate.c:1374:osp_precreate_thread()) Skipped 2 previous similar messages Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 414 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages LustreError: 2672:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712691593 with bad export cookie 16058118230310799381 LustreError: 166-1: MGC192.168.123.41@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages | Link to test |
| conf-sanity test 100: check lshowmount lists MGS, MDT, OST and 0@lo | BUG: unable to handle kernel paging request at ffff88007b4802f8 IP: [<ffffffffa137b23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fc02067 PMD 33fa27067 PTE 800000007b480060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata floppy virtio_blk serio_raw i2c_core [last unloaded: libcfs] CPU: 2 PID: 9646 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031ab60010 ti: ffff8802c799c000 task.ti: ffff8802c799c000 RIP: 0010:[<ffffffffa137b23d>] [<ffffffffa137b23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802c799fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88009fdca2a8 RCX: ffff88028326203a RDX: 0000000000000002 RSI: ffff88007b4802f0 RDI: ffff8802be65b8e8 RBP: ffff8802c799fce0 R08: ffff88028326204a R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88007b480000 R13: ffff8802c799fcf0 R14: ffffffffa13e12c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88007b4802f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa04135a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa04139e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0415605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa042abee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa042b7e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa040f068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa040f4f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-34.localnet: executing set_hostid alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.36@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space. Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 23701:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712606535 with bad export cookie 9872579758264970214 LustreError: 166-1: MGC192.168.123.36@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages | Link to test |
| conf-sanity test 73: failnode to update from mountdata properly | BUG: unable to handle kernel paging request at ffff8802c51d82f8 IP: [<ffffffffa138323d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e9d0067 PTE 80000002c51d8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs] CPU: 6 PID: 13365 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c99ddc40 ti: ffff8800a8b74000 task.ti: ffff8800a8b74000 RIP: 0010:[<ffffffffa138323d>] [<ffffffffa138323d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8800a8b77cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880097fee7e8 RCX: ffff8802c2a5c125 RDX: 0000000000000006 RSI: ffff8802c51d82f0 RDI: ffff8802d33b95a8 RBP: ffff8800a8b77ce0 R08: ffff8802c2a5c135 R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802c51d8000 R13: ffff8800a8b77cf0 R14: ffffffffa13e92c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802c51d82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa038a5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa038a9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa038c605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a1bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03a27e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0386068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03864f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.16@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Found index 0 for lustre-OST0000, updating log Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 6354:0:(import.c:353:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) LustreError: 6354:0:(import.c:377:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff88008b8bbc40 x1795617230709184/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1712434084 ref 2 fl UnregRPC:ENU/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 LustreError: 6354:0:(import.c:387:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 3599:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712434113 with bad export cookie 10084913451862064051 LustreError: 166-1: MGC192.168.123.16@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff88029a4902f8 IP: [<ffffffffa139023d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb27067 PTE 800000029a490060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk libata floppy i2c_core [last unloaded: libcfs] CPU: 1 PID: 27334 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802a6a049d0 ti: ffff880320b10000 task.ti: ffff880320b10000 RIP: 0010:[<ffffffffa139023d>] [<ffffffffa139023d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880320b13cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802fa91b3f8 RCX: ffff8800aeb64c07 RDX: 0000000000000001 RSI: ffff88029a4902f0 RDI: ffff88029f92c1b8 RBP: ffff880320b13ce0 R08: ffff8800aeb64c17 R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88029a490000 R13: ffff880320b13cf0 R14: ffffffffa13f62c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88029a4902f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa040f5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa040f9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0411605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa0426bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa04277e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa040b068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa040b4f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:35 to 0x240000400:97 Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 4304:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff880249f06e40 x1795508615006656/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1712330596 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client LustreError: 4304:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 4304:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 4304:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 | Link to test |
| conf-sanity test 39: leak_finder recognizes both LUSTRE and LNET malloc messages | BUG: unable to handle kernel paging request at ffff88029d4182f8 IP: [<ffffffffa139b078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb0f067 PTE 800000029d418060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks ata_piix floppy virtio_blk libata serio_raw i2c_core [last unloaded: libcfs] CPU: 13 PID: 1736 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88024437b760 ti: ffff8802bff8c000 task.ti: ffff8802bff8c000 RIP: 0010:[<ffffffffa139b078>] [<ffffffffa139b078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802bff8fcc8 EFLAGS: 00010246 RAX: ffffffffa13f2800 RBX: ffff88029b6cd698 RCX: ffff8800abbf83fe RDX: 0000000000000001 RSI: ffff88029d4182f0 RDI: ffff8802bff8fcf0 RBP: ffff8802bff8fce0 R08: ffff8800abbf840b R09: 74696e695f79656b R10: 0000000000000000 R11: 000000000000000f R12: ffff88029d418000 R13: ffff8802bff8fcf0 R14: ffffffffa14012c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88029d4182f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03d55a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03d59e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03d7605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ecbee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ed7e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03d1068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03d14f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_230_g3bbb0f7 LNet: Added LNI 192.168.123.121@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:129 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000bd0:35 to 0x2c0000bd0:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 13297:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1712323078 with bad export cookie 2505601049062779596 LustreError: 166-1: MGC192.168.123.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Skipped 1 previous similar message | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff88029de402f8 IP: [<ffffffffa138023d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb0a067 PTE 800000029de40060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk i2c_core libata floppy [last unloaded: libcfs] CPU: 0 PID: 18085 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800a789dc40 ti: ffff880295120000 task.ti: ffff880295120000 RIP: 0010:[<ffffffffa138023d>] [<ffffffffa138023d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880295123cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880097ffd698 RCX: ffff88024c5ad89f RDX: 0000000000000000 RSI: ffff88029de402f0 RDI: ffff88009557ca88 RBP: ffff880295123ce0 R08: ffff88024c5ad8af R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88029de40000 R13: ffff880295123cf0 R14: ffffffffa13e62c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88029de402f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa039b5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa039b9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa039d605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b2bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b37e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0397068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03974f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:35 to 0x240000400:97 Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 21346:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff880093ef7840 x1794926815757696/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1711775735 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client LustreError: 21346:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 21346:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 21346:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) | Link to test |
| replay-dual test 26: dbench and tar with mds failover | BUG: unable to handle kernel paging request at ffff88025acf02f8 IP: [<ffffffffa138123d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ed24067 PTE 800000025acf0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw i2c_core floppy libata [last unloaded: libcfs] CPU: 4 PID: 1919 Comm: kworker/u32:6 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802b495dc40 ti: ffff880088b78000 task.ti: ffff880088b78000 RIP: 0010:[<ffffffffa138123d>] [<ffffffffa138123d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880088b7bcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88009d630008 RCX: ffff88028735d471 RDX: 0000000000000004 RSI: ffff88025acf02f0 RDI: ffff8802f43c55d8 RBP: ffff880088b7bce0 R08: ffff88028735d481 R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88025acf0000 R13: ffff880088b7bcf0 R14: ffffffffa13e72c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88025acf02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa038b5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa038b9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa038d605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a2bee>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03a37e8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0387068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03874f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 21 previous similar messages Lustre: 25414:0:(mdt_recovery.c:148:mdt_req_from_lrd()) @@@ restoring transno req@ffff880094cfa840 x1794699112652544/t154618822778(0) o36->7d9a0f09-8dd8-49ef-b761-6975cfe03359@0@lo:390/0 lens 488/3152 e 0 to 0 dl 1711560475 ref 1 fl Interpret:/202/0 rc 0/0 job:'tar.0' uid:0 gid:0 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1925 to 0x300000403:1953 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1573 to 0x340000403:1601 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1574 to 0x2c0000404:1601 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:1574 to 0x380000404:1601 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 0: single mount setup | BUG: unable to handle kernel paging request at ffff8802d60c02f8 IP: [<ffffffffa137623d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e948067 PTE 80000002d60c0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw i2c_core virtio_blk libata floppy [last unloaded: libcfs] CPU: 6 PID: 8850 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88007a7c3760 ti: ffff8802a1180000 task.ti: ffff8802a1180000 RIP: 0010:[<ffffffffa137623d>] [<ffffffffa137623d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802a1183cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88021bea67e8 RCX: ffff8802c1922e2c RDX: 0000000000000006 RSI: ffff8802d60c02f0 RDI: ffff880263f5bae8 RBP: ffff8802a1183ce0 R08: ffff8802c1922e3c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802d60c0000 R13: ffff8802a1183cf0 R14: ffffffffa13dc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802d60c02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03bb5a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03bb9e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03bd605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d2ace>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d36c8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b7068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b74f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 5177:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711030404 with bad export cookie 6084770085402825566 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 5177:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 6 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 8918:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 10055:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711030502 with bad export cookie 9910028148900592801 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 13875:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:67 to 0x2c0000402:97 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 15013:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711030580 with bad export cookie 3965016959426445423 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 18997:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:99 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 22561:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711030662 with bad export cookie 4633888375744787151 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 22561:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: 19427:0:(client.c:2340:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1711030656/real 1711030656] req@ffff8802852eb240 x1794145629790400/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1711030672 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 24357:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:131 to 0x2c0000402:161 Lustre: Mounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 25521:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711030766 with bad export cookie 14485668446744593022 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 29678:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:163 to 0x2c0000402:193 Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 30823:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711030850 with bad export cookie 5643454494700186316 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0002-lwp-OST0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete Key type lgssc unregistered LNet: 2615:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 3719:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.6@tcp: failed processing log, type 1: rc = -5 LustreError: 3719:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.6@tcp: failed processing log, type 4: rc = -110 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 15c-8: MGC192.168.123.6@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info Lustre: Unmounted lustre-client LustreError: 4182:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -5 Lustre: server umount lustre-OST0000 complete Key type lgssc unregistered LNet: 4718:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 8411:0:(llite_lib.c:1416:ll_fill_super()) wrong.lustre: fsname longer than 8 characters: rc = -36 Lustre: Unmounted wrong.lustre-client LustreError: 8411:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -36 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 5875:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711031329 with bad export cookie 16298691189752642351 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 9333:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 10477:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.6@tcp: failed processing log, type 1: rc = -5 LustreError: 10477:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.6@tcp: failed processing log, type 4: rc = -110 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:225 LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: Mounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages LustreError: 10500:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711031613 with bad export cookie 367721346979048494 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 14561:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:257 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 15000:0:(niobuf.c:950:ptl_send_rpc()) cfs_fail_timeout id 506 sleeping for 16000ms Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 15000:0:(niobuf.c:950:ptl_send_rpc()) cfs_fail_timeout id 506 awake LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 15719:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711031680 with bad export cookie 16916047949926872070 LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 19598:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.6@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_201_g1218e31 LNet: Added LNI 192.168.123.6@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-4.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:289 Lustre: Mounted lustre-client Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: *** cfs_fail_loc=135, val=0*** LustreError: 24656:0:(obd_config.c:797:class_setup()) setup lustre-MDT0000 failed (-2) LustreError: 24656:0:(obd_config.c:2037:class_config_llog_handler()) MGC192.168.123.6@tcp: cfg command failed: rc = -2 Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f LustreError: 15c-8: MGC192.168.123.6@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info LustreError: 24611:0:(tgt_mount.c:1527:server_start_targets()) failed to start server lustre-MDT0000: -2 LustreError: 24611:0:(tgt_mount.c:2223:server_fill_super()) Unable to start targets: -2 LustreError: 24611:0:(obd_config.c:850:class_cleanup()) Device 5 not setup LustreError: 24021:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711031752 with bad export cookie 10994190284944265376 LustreError: 24021:0:(ldlm_lock.c:2823:ldlm_lock_dump_handle()) ### ### ns: MGS lock: ffff8802c935a200/0x989335d07f2750bc lrc: 3/0,0 mode: CR/CR res: [0x65727473756c:0x0:0x0].0x0 rrc: 2 type: PLN flags: 0x40000000000000 nid: 0@lo remote: 0x989335d07f2750b5 expref: 3 pid: 24616 timeout: 0 lvb_type: 0 Lustre: server umount lustre-MDT0000 complete LustreError: 24611:0:(super25.c:189:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -2 LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.6@tcp (at 0@lo) Lustre: lustre-MDT0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:321 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 20992:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1711031781 with bad export cookie 10994190284944265516 LustreError: 20992:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 166-1: MGC192.168.123.6@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 11 previous similar messages | Link to test |
| conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount | BUG: unable to handle kernel paging request at ffff88008d8082f8 IP: [<ffffffffa137223d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f994067 PTE 800000008d808060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy libata i2c_core [last unloaded: libcfs] CPU: 9 PID: 8235 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008d13a4f0 ti: ffff8802bb93c000 task.ti: ffff8802bb93c000 RIP: 0010:[<ffffffffa137223d>] [<ffffffffa137223d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802bb93fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8800a9011158 RCX: ffff8802981297b4 RDX: 0000000000000009 RSI: ffff88008d8082f0 RDI: ffff8802b3af29c8 RBP: ffff8802bb93fce0 R08: ffff8802981297c4 R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88008d808000 R13: ffff8802bb93fcf0 R14: ffffffffa13d82c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88008d8082f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03875a4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03879e0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0389605>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa039eace>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa039f6c8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0383068>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03834f2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-69.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 13367:0:(lod_sub_object.c:966:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 LustreError: 13367:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space. Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: 14361:0:(lod_sub_object.c:966:lod_sub_prep_llog()) lustre-MDT0002-mdtlov: can't create new llog: rc = -115 LustreError: 14361:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0002-osd: get update log duration 0, retries 0, failed: rc = -115 Lustre: 14362:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802a942ad40 x1793482184224000/t0(0) o700->lustre-MDT0001-osp-MDT0002@0@lo:30/10 lens 264/248 e 0 to 0 dl 1710398004 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space. Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: MGS: Client ff85a62e-91f8-4f5a-bdfa-30d0404077cb (at 0@lo) reconnecting Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 LustreError: 12290:0:(mgs_handler.c:264:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message LustreError: 13204:0:(mgs_handler.c:264:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 Lustre: 15812:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802c6d84140 x1793482184230912/t0(0) o700->lustre-OST0000-osc-MDT0002@0@lo:31/4 lens 264/248 e 0 to 0 dl 1710398011 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 16309:0:(client.c:1513:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802e19f2840 x1793482184231552/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1710398012 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: 16309:0:(client.c:1513:after_reply()) Skipped 3 previous similar messages Lustre: srv-lustre-MDT0002: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:2:mdt] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000400 Lustre: Mounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 17267:0:(file.c:264:ll_close_inode_openhandle()) lustre-clilmv-ffff8802a6bb9bf8: inode [0x200000403:0x1:0x0] mdc close failed: rc = -108 Lustre: 900:0:(llite_lib.c:4078:ll_dirty_page_discard_warn()) lustre: dirty page discard: 192.168.123.71@tcp:/lustre/fid: [0x200000403:0x1:0x0]/ may get corrupted (rc -108) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0001 complete | Link to test |
| conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced | BUG: unable to handle kernel paging request at ffff8800a70f82f8 IP: [<ffffffffa137123d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f8c8067 PTE 80000000a70f8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs] CPU: 5 PID: 2783 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c6a549d0 ti: ffff8802cdf10000 task.ti: ffff8802cdf10000 RIP: 0010:[<ffffffffa137123d>] [<ffffffffa137123d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802cdf13cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802a12333f8 RCX: ffff8802c6a5a125 RDX: 0000000000000005 RSI: ffff8800a70f82f0 RDI: ffff8802e45c4d08 RBP: ffff8802cdf13ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8800a70f8000 R13: ffff8802cdf13cf0 R14: ffffffffa13d72c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800a70f82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03aa574>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03aa9b0>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03ac5d5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c1a9e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c2698>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a6038>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a64c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Skipped 1 previous similar message Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 5549:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002081/real 1710002081] req@ffff8802c43a1940 x1793065650836608/t0(0) o36->lustre-MDT0000-mdc-ffff8802c2388958@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002097 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 5549:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: lustre-MDT0000: Client 53c08c03-b8fc-4d70-9288-82507fe3ee5d (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8802c2388958: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 5 previous similar messages Lustre: 5619:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002097/real 1710002097] req@ffff8802eb322340 x1793065650846016/t0(0) o36->lustre-MDT0000-mdc-ffff8802c2388958@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002113 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Client 53c08c03-b8fc-4d70-9288-82507fe3ee5d (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8802c2388958: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: Unmounted lustre-client Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete Lustre: 6102:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002123/real 1710002123] req@ffff8802754cbc40 x1793065650860928/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1710002133 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 6673:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.121@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_141_ga511785 LNet: Added LNI 192.168.123.121@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:42 to 0x2c0000402:65 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 10780:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002184/real 1710002184] req@ffff8802c40e8a40 x1793067215508864/t0(0) o36->lustre-MDT0000-mdc-ffff8802906c5d28@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002200 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000-mdc-ffff8802906c5d28: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Client c9232e1b-35f8-4308-80f3-a026dba6e3b0 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8802906c5d28: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 10931:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002201/real 1710002201] req@ffff8802e6049940 x1793067215522560/t0(0) o36->lustre-MDT0001-mdc-ffff8802906c5d28@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002217 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0001-mdc-ffff8802906c5d28: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Client c9232e1b-35f8-4308-80f3-a026dba6e3b0 (at 0@lo) reconnecting Lustre: lustre-MDT0001-mdc-ffff8802906c5d28: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 3 previous similar messages Lustre: 11132:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002217/real 1710002217] req@ffff880324212340 x1793067215531264/t0(0) o36->lustre-MDT0001-mdc-ffff8802906c5d28@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002233 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0001-mdc-ffff8802906c5d28: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Client c9232e1b-35f8-4308-80f3-a026dba6e3b0 (at 0@lo) reconnecting Lustre: lustre-MDT0001-mdc-ffff8802906c5d28: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 4 previous similar messages Lustre: 11445:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002235/real 1710002235] req@ffff8800688a7340 x1793067215555264/t0(0) o36->lustre-MDT0002-mdc-ffff8802913a53d8@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002251 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0002-mdc-ffff8802913a53d8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: 11445:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: lustre-MDT0002: Client 9e3f2677-d8a4-4901-aa28-8dd28b2073f5 (at 0@lo) reconnecting Lustre: lustre-MDT0002-mdc-ffff8802913a53d8: Connection restored to (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 14 previous similar messages Lustre: 11719:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002251/real 1710002251] req@ffff8802b4c94640 x1793067215568256/t0(0) o36->lustre-MDT0002-mdc-ffff8802913a53d8@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002267 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 11719:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: lustre-MDT0002-mdc-ffff8802913a53d8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0002: Client 9e3f2677-d8a4-4901-aa28-8dd28b2073f5 (at 0@lo) reconnecting Lustre: lustre-MDT0002-mdc-ffff8802913a53d8: Connection restored to (at 0@lo) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation out_update to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: 12017:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002273/real 1710002273] req@ffff88008ce95a40 x1793067215593984/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1710002283 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 7810:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1710002285 with bad export cookie 17366570331447959768 LustreError: 166-1: MGC192.168.123.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 12818:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.121@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_141_ga511785 LNet: Added LNI 192.168.123.121@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:68 to 0x2c0000402:97 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:72 to 0x2c0000401:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:83 to 0x2c0000400:129 Lustre: Unmounted lustre-client Lustre: Mounted lustre-client LustreError: 17288:0:(genops.c:2121:obd_set_max_mod_rpcs_in_flight()) lustre-MDT0000-mdc-ffff88007325ca88: can't set max_mod_rpcs_in_flight=9 higher than mdt.*.max_mod_rpcs_in_flight=8 returned by the MDT server at connection. Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params Lustre: Skipped 1 previous similar message Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 13971:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1710002347 with bad export cookie 196091399544927965 LustreError: 166-1: MGC192.168.123.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: 17882:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002347/real 1710002347] req@ffff8802a95d9440 x1793067373856128/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1710002357 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 18498:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.121@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_141_ga511785 LNet: Added LNI 192.168.123.121@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:72 to 0x2c0000401:129 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:83 to 0x2c0000400:161 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:68 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 22702:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1710002418/real 1710002418] req@ffff880283339440 x1793067450395392/t0(0) o36->lustre-MDT0000-mdc-ffff8800853d6fc8@0@lo:12/10 lens 488/512 e 0 to 1 dl 1710002434 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000-mdc-ffff8800853d6fc8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Client 035135f8-f463-4044-85cb-8572d3400638 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8800853d6fc8: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: 22702:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 2 previous similar messages Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 19632:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1710002452 with bad export cookie 16684006039167691145 LustreError: 166-1: MGC192.168.123.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 23748:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.121@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_141_ga511785 LNet: Added LNI 192.168.123.121@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:83 to 0x2c0000400:193 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:138 to 0x2c0000402:161 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:72 to 0x2c0000401:161 Lustre: Mounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: lustre-OST0000: evicting d432e665-4001-48f2-ab89-e377803f1947 (at 0@lo) by administrative request Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 167-0: lustre-OST0000-osc-MDT0002: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0002: Connection restored to 192.168.123.121@tcp (at 0@lo) LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.123.121@tcp (at 0@lo) LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.121@tcp (at 0@lo) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 24923:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1710002564 with bad export cookie 15898586545659103182 LustreError: 166-1: MGC192.168.123.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 24923:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 29225:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.121@tcp alg: No test for adler32 (adler32-zlib) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 Lustre: Lustre: Build Version: 2.15.61_141_ga511785 LNet: Added LNI 192.168.123.121@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 LustreError: 4267:0:(mgs_llog.c:4580:mgs_write_log_target()) cfs_fail_timeout id 90e sleeping for 10000ms LustreError: 4267:0:(mgs_llog.c:4580:mgs_write_log_target()) cfs_fail_timeout id 90e awake Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 12 sec Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:2:mdt] Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-119.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88008f69b7e8.ost_server_uuid 50 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88008f69b7e8.ost_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 7853:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1710002683 with bad export cookie 7621012984637962232 LustreError: 166-1: MGC192.168.123.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 7853:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 0: single mount setup | BUG: unable to handle kernel paging request at ffff8802ed6a82f8 IP: [<ffffffffa135f23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e88d067 PTE 80000002ed6a8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm drm_panel_orientation_quirks ata_piix serio_raw i2c_core libata virtio_blk floppy [last unloaded: libcfs] CPU: 9 PID: 19082 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880325d1dc40 ti: ffff8802b1704000 task.ti: ffff8802b1704000 RIP: 0010:[<ffffffffa135f23d>] [<ffffffffa135f23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802b1707cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8800a8f20008 RCX: ffff8802910c8c65 RDX: 0000000000000009 RSI: ffff8802ed6a82f0 RDI: ffff8802cbd2cf08 RBP: ffff8802b1707ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802ed6a8000 R13: ffff8802b1707cf0 R14: ffffffffa13c52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802ed6a82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a5544>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a5980>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a75a5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03bca0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03bd608>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a1038>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a14c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 11413:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709577735 with bad export cookie 13795171870283401069 LustreError: 11413:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 15164:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_120_g32ad531 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 16291:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709577814 with bad export cookie 11248611241686390200 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 20072:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_120_g32ad531 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:67 to 0x2c0000402:97 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 21160:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709577895 with bad export cookie 13075659643117488619 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: 20503:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709577888/real 1709577888] req@ffff8802e45cf840 x1792622292132352/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1709577904 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages Lustre: 24637:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1709577895/real 1709577895] req@ffff8802796f8a40 x1792622292134272/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1709577905 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 25228:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_120_g32ad531 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:99 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 26399:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709577969 with bad export cookie 1825597877013985608 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 30272:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_120_g32ad531 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:131 to 0x2c0000402:161 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 31413:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709578059 with bad export cookie 1062585395205590537 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 3274:0:(lib-ptl.c:990:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_120_g32ad531 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:163 to 0x2c0000402:193 Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-MDT0000 complete LustreError: 4443:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709578138 with bad export cookie 11161202169040231473 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message | Link to test |
| replay-dual test 26: dbench and tar with mds failover | BUG: unable to handle kernel paging request at ffff88026d8002f8 IP: [<ffffffffa136a23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ec8e067 PTE 800000026d800060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks virtio_blk ata_piix i2c_core serio_raw libata floppy [last unloaded: libcfs] CPU: 7 PID: 20493 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031ada9280 ti: ffff88028b5fc000 task.ti: ffff88028b5fc000 RIP: 0010:[<ffffffffa136a23d>] [<ffffffffa136a23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88028b5ffcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88023aa0a2a8 RCX: ffff8802f4537d5b RDX: 0000000000000007 RSI: ffff88026d8002f0 RDI: ffff880293737348 RBP: ffff88028b5ffce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88026d800000 R13: ffff88028b5ffcf0 R14: ffffffffa13d02c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88026d8002f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0406544>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0406980>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa04085a5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa041da0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa041e608>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0402038>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa04024c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:1671 to 0x340000404:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:1671 to 0x380000404:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1672 to 0x300000403:1697 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1672 to 0x2c0000404:1697 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: Failing over lustre-MDT0001 LustreError: 26043:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709274096 with bad export cookie 17954355676306205712 LustreError: 26043:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 201 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:249 to 0x380000405:289 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:249 to 0x300000405:289 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:250 to 0x340000405:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:249 to 0x2c0000405:289 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times Lustre: Failing over lustre-MDT0002 LustreError: 11-0: lustre-MDT0002-mdc-ffff880066941bf8: operation ldlm_enqueue to node 0@lo failed: rc = -19 LustreError: Skipped 30 previous similar messages Lustre: lustre-MDT0002-mdc-ffff880066941bf8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 205 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:34 to 0x380000403:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:27 to 0x2c0000403:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:32 to 0x340000403:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:32 to 0x300000404:65 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1718 to 0x300000403:1761 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:1721 to 0x340000404:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1714 to 0x2c0000404:1729 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:1717 to 0x380000404:1761 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: Failing over lustre-MDT0001 | Link to test |
| recovery-small test 136: changelog_deregister leaving pending records | BUG: unable to handle kernel paging request at ffff8802ec5402f8 IP: [<ffffffffa135b23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e896067 PTE 80000002ec540060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix floppy drm_panel_orientation_quirks virtio_blk i2c_core serio_raw libata [last unloaded: libcfs] CPU: 1 PID: 2135 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031cbd0010 ti: ffff88029ef58000 task.ti: ffff88029ef58000 RIP: 0010:[<ffffffffa135b23d>] [<ffffffffa135b23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88029ef5bcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880286919158 RCX: ffff88028fe1ab8c RDX: 0000000000000001 RSI: ffff8802ec5402f0 RDI: ffff88028b9f0088 RBP: ffff88029ef5bce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802ec540000 R13: ffff88029ef5bcf0 R14: ffffffffa13c12c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802ec5402f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0396544>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0396980>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03985a5>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ada0e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ae608>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0392038>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03924c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDD0000: changelog on Lustre: DEBUG MARKER: recovery-small test_136: @@@@@@ FAIL: Restart of ost4 failed! Lustre: DEBUG MARKER: centos-107.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST0 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 6 previous similar messages Lustre: Unmounted lustre-client Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 25886:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1709150541 with bad export cookie 7571703366146722023 LustreError: 166-1: MGC192.168.123.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 1 previous similar message LustreError: 25886:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 8 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages | Link to test |
| replay-ost-single test complete, duration 821 sec | BUG: unable to handle kernel paging request at ffff8802a41002f8 IP: [<ffffffffa134f078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ead9067 PTE 80000002a4100060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks serio_raw virtio_blk floppy libata i2c_core [last unloaded: libcfs] CPU: 4 PID: 27915 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802be61b760 ti: ffff8802d9720000 task.ti: ffff8802d9720000 RIP: 0010:[<ffffffffa134f078>] [<ffffffffa134f078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802d9723cc8 EFLAGS: 00010246 RAX: ffffffffa13a6820 RBX: ffff88029f57a2a8 RCX: ffff8802b8b9e568 RDX: 0000000000000001 RSI: ffff8802a41002f0 RDI: ffff8802d9723cf0 RBP: ffff8802d9723ce0 R08: ffff88027647dfa8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000004 R12: ffff8802a4100000 R13: ffff8802d9723cf0 R14: ffffffffa13b62c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802a41002f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b9f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ba350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03bbf75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d13be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d1d38>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b5a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b5e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 6 previous similar messages Lustre: Unmounted lustre-client Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages LustreError: 23095:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708947756 with bad export cookie 16089719975093898418 LustreError: 166-1: MGC192.168.123.23@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 23095:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages | Link to test |
| replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 | BUG: unable to handle kernel paging request at ffff8802a26a82f8 IP: [<ffffffffa1363078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eae6067 PTE 80000002a26a8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs] CPU: 15 PID: 1184 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88023acba4f0 ti: ffff880241afc000 task.ti: ffff880241afc000 RIP: 0010:[<ffffffffa1363078>] [<ffffffffa1363078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880241affcc8 EFLAGS: 00010246 RAX: ffffffffa13ba820 RBX: ffff8803229eb3f8 RCX: ffff88026b534fd8 RDX: 0000000000000001 RSI: ffff8802a26a82f0 RDI: ffff880241affcf0 RBP: ffff880241affce0 R08: ffff88007c12fa38 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000f R12: ffff8802a26a8000 R13: ffff880241affcf0 R14: ffffffffa13ca2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802a26a82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b8f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b9350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03baf75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d03be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d0d38>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b4a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b4e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 87 previous similar messages | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff8802f5e402f8 IP: [<ffffffffa136a23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e849067 PTE 80000002f5e40060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ttm drm ata_generic pata_acpi drm_panel_orientation_quirks ata_piix virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs] CPU: 5 PID: 32594 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880093a849d0 ti: ffff880280b34000 task.ti: ffff880280b34000 RIP: 0010:[<ffffffffa136a23d>] [<ffffffffa136a23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880280b37cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff8802cd079158 RCX: ffff8802b1e91829 RDX: 0000000000000005 RSI: ffff8802f5e402f0 RDI: ffff880081873a68 RBP: ffff880280b37ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802f5e40000 R13: ffff880280b37cf0 R14: ffffffffa13d12c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802f5e402f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a5f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a6350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a7f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03bd3be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03bdd38>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a1a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a1e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 30282:0:(client.c:1511:after_reply()) @@@ resending request on EINPROGRESS req@ffff88006dce1440 x1791383067235328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1708396207 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 13 previous similar messages LustreError: 30282:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 30282:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 30282:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 Lustre: server umount fs15246-MDT0000 complete Lustre: server umount fs15246-OST0000 complete Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 31237:0:(import.c:353:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) LustreError: 31237:0:(import.c:377:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880254a5c640 x1791383067245184/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1708396211 ref 2 fl UnregRPC:ENU/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 LustreError: 31237:0:(import.c:387:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. LustreError: 31237:0:(import.c:353:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) LustreError: 31237:0:(import.c:377:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880254a5c640 x1791383067245184/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1708396211 ref 2 fl UnregRPC:ENU/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 LustreError: 31237:0:(import.c:387:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 23 previous similar messages LustreError: 31237:0:(import.c:353:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) LustreError: 31237:0:(import.c:377:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880254a5c640 x1791383067245184/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1708396211 ref 2 fl UnregRPC:ENU/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 LustreError: 31237:0:(import.c:387:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. Lustre: server umount lustre-MDT0000 complete LustreError: 24066:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708396255 with bad export cookie 16399787189705997281 LustreError: 24066:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 32140:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_78_gf8007dd LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: centos-49.localnet: executing set_hostid Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Not available for connect from 0@lo (not set up) Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 9179:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708396450 with bad export cookie 11136425318653874567 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 117: lctl get_param return errors properly | BUG: unable to handle kernel paging request at ffff8802e3ec02f8 IP: [<ffffffffa137623d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e8d9067 PTE 80000002e3ec0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk libata serio_raw i2c_core [last unloaded: libcfs] CPU: 3 PID: 27733 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800b4e2b760 ti: ffff88025c18c000 task.ti: ffff88025c18c000 RIP: 0010:[<ffffffffa137623d>] [<ffffffffa137623d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88025c18fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88006a43b3f8 RCX: ffff8802ac4e1942 RDX: 0000000000000003 RSI: ffff8802e3ec02f0 RDI: ffff880092b7e7f8 RBP: ffff88025c18fce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802e3ec0000 R13: ffff88025c18fcf0 R14: ffffffffa13dd2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802e3ec02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a5f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a6350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a7f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03bd3be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03bdd38>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a1a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a1e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 4 previous similar messages Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-MDT0000 complete LustreError: 24947:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708382605 with bad export cookie 12745131059742798009 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 24947:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 8 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: conf-sanity test_117: @@@@@@ FAIL: cleanup failed with rc 203 Lustre: 2500:0:(client.c:2338:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1708382615/real 1708382615] req@ffff88008afd6440 x1791368686746752/t0(0) o400->lustre-MDT0002-lwp-OST0003@0@lo:12/10 lens 224/224 e 0 to 1 dl 1708382631 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: 2500:0:(client.c:2338:ptlrpc_expire_one_request()) Skipped 4 previous similar messages Lustre: lustre-MDT0002-lwp-OST0003: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: server umount lustre-OST0001 complete Lustre: server umount lustre-OST0002 complete Lustre: server umount lustre-OST0003 complete Key type lgssc unregistered LNet: 7624:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.61_78_gf8007dd LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 14953:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708382757 with bad export cookie 905744521188534223 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages | Link to test |
| racer test complete, duration 2904 sec | BUG: unable to handle kernel paging request at ffff880251b602f8 IP: [<ffffffffa135b078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ed6d067 PTE 8000000251b60060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw floppy i2c_core libata [last unloaded: libcfs] CPU: 7 PID: 3085 Comm: kworker/u32:6 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880265898010 ti: ffff880079afc000 task.ti: ffff880079afc000 RIP: 0010:[<ffffffffa135b078>] [<ffffffffa135b078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880079affcc8 EFLAGS: 00010246 RAX: ffffffffa13b2820 RBX: ffff88023108b3f8 RCX: ffff8802845273c8 RDX: 0000000000000004 RSI: ffff880251b602f0 RDI: ffff880079affcf0 RBP: ffff880079affce0 R08: ffff8800a850cfb8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000007 R12: ffff880251b60000 R13: ffff880079affcf0 R14: ffffffffa13c22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880251b602f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0394f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0395350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0396f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ac3be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03acd38>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0390a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa0390e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 6 previous similar messages Lustre: Unmounted lustre-client Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: haven't heard from client 19189012-7b34-48e0-8a77-49fa3cdd3fd1 (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp ffff880289f12548, cur 1708287093 expire 1708287063 last 1708287061 LustreError: 3680:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708287094 with bad export cookie 17052770575649938287 LustreError: 166-1: MGC192.168.123.67@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test |
| conf-sanity test 80: mgc import reconnect race | BUG: unable to handle kernel paging request at ffff8802bffd82f8 IP: [<ffffffffa137123d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33e9fa067 PTE 80000002bffd8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) osp(OE) ofd(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw i2c_core virtio_blk libata floppy [last unloaded: libcfs] CPU: 13 PID: 5620 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031a051280 ti: ffff880327b9c000 task.ti: ffff880327b9c000 RIP: 0010:[<ffffffffa137123d>] [<ffffffffa137123d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880327b9fcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880325fea2a8 RCX: ffff8800b48c1829 RDX: 000000000000000d RSI: ffff8802bffd82f0 RDI: ffff8800b247d108 RBP: ffff880327b9fce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802bffd8000 R13: ffff880327b9fcf0 R14: ffffffffa13d82c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802bffd82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c9f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ca350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03cbf75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03e13be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03e1d38>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03c5a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c5e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: 9153:0:(genops.c:1641:obd_export_evict_by_uuid()) MGS: evicting fe42c54a-8153-4b45-b0df-7b93eafe2a18 at adminstrative request LustreError: 11-0: MGC192.168.123.16@tcp: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: 166-1: MGC192.168.123.16@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: 6674:0:(mgc_request.c:1825:mgc_process_log()) MGC192.168.123.16@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 Lustre: *** cfs_fail_loc=906, val=2147483648*** Lustre: MGS: Client fe42c54a-8153-4b45-b0df-7b93eafe2a18 (at 0@lo) reconnecting Lustre: Evicted from MGS (at 192.168.123.16@tcp) after server handle changed from 0xf6ae61628dcc7626 to 0xf6ae61628dcc79de Lustre: MGC192.168.123.16@tcp: Connection restored to 192.168.123.16@tcp (at 0@lo) Lustre: lustre-OST0001: new disk, initializing Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: server umount lustre-OST0001 complete Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 10172:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708181201 with bad export cookie 17775251855191603678 LustreError: 166-1: MGC192.168.123.16@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 10172:0:(ldlm_lockd.c:2594:ldlm_cancel_handler()) Skipped 8 previous similar messages | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88009d0102f8 IP: [<ffffffffa136c078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f918067 PTE 800000009d010060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy serio_raw i2c_core libata virtio_blk [last unloaded: libcfs] CPU: 14 PID: 14973 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880319775c40 ti: ffff8802ce4d0000 task.ti: ffff8802ce4d0000 RIP: 0010:[<ffffffffa136c078>] [<ffffffffa136c078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802ce4d3cc8 EFLAGS: 00010246 RAX: ffffffffa13c2820 RBX: ffff8802838f33f8 RCX: ffff880073a24cf8 RDX: 0000000000000001 RSI: ffff88009d0102f0 RDI: ffff8802ce4d3cf0 RBP: ffff8802ce4d3ce0 R08: ffff8802668a9fe8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000e R12: ffff88009d010000 R13: ffff8802ce4d3cf0 R14: ffffffffa13d22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88009d0102f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c6f24>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c7360>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c8f85>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03de3ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ded28>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03c2a18>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c2ea2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=15701 ... LustreError: 17107:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 17107:0:(osd_handler.c:698:osd_ro()) Skipped 13 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 138 previous similar messages LustreError: 15985:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0000-mdc-ffff880282b29bf8: can't stat MDS #0: rc = -19 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3930 to 0x300000403:3969 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3933 to 0x2c0000403:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3930 to 0x380000403:3969 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 14663:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008754d540 x1791002481685056/t4294967781(4294967781) o101->lustre-MDT0001-mdc-ffff880282b29bf8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1708036305 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 14663:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:59 to 0x2c0000405:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:58 to 0x300000405:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:59 to 0x380000405:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:58 to 0x340000404:129 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:45 to 0x2c0000404:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:3 to 0x380000404:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:4 to 0x300000404:65 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 15985:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0000-mdc-ffff880282b29bf8: can't stat MDS #0: rc = -19 LustreError: 15985:0:(lmv_obd.c:1383:lmv_statfs()) Skipped 2 previous similar messages LustreError: 14663:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802c60dd540 x1791002484001728/t0(0) o250->MGC192.168.123.8@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3930 to 0x380000403:4001 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:4001 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3930 to 0x300000403:4001 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3933 to 0x2c0000403:4001 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001-mdc-ffff880282b29bf8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 65 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 14663:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008754d540 x1791002481685056/t4294967781(4294967781) o101->lustre-MDT0001-mdc-ffff880282b29bf8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1708036383 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 14663:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-MDT0001-lwp-OST0001: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 70 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:281 to 0x340000404:321 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:280 to 0x2c0000405:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:281 to 0x380000405:321 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:281 to 0x300000405:321 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 | Link to test |
| conf-sanity test 57a: initial registration from failnode should fail (should return errs) | BUG: unable to handle kernel paging request at ffff8802917282f8 IP: [<ffffffffa135b23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb6e067 PTE 8000000291728060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk libata i2c_core [last unloaded: libcfs] CPU: 14 PID: 29369 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008e21dc40 ti: ffff8802c1db8000 task.ti: ffff8802c1db8000 RIP: 0010:[<ffffffffa135b23d>] [<ffffffffa135b23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802c1dbbcc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff880320694548 RCX: ffff8802eb23d1a3 RDX: 000000000000000e RSI: ffff8802917282f0 RDI: ffff8802bf8db598 RBP: ffff8802c1dbbce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880291728000 R13: ffff8802c1dbbcf0 R14: ffffffffa13c12c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802917282f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b6f24>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b7360>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b8f85>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ce3ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ced28>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b2a18>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b2ea2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-39.localnet: executing load_modules_local Lustre: DEBUG MARKER: centos-39.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 2 previous similar messages Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: new disk, initializing Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Skipped 7 previous similar messages Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 16256:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708033841 with bad export cookie 9020486199798693186 LustreError: 16256:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.41@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Denying initial registration attempt from nid 192.168.123.41@tcp, specified as failover LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. LustreError: 24931:0:(tgt_mount.c:2216:server_fill_super()) Unable to start targets: -99 LustreError: 24931:0:(tgt_mount.c:1752:server_put_super()) no obd lustre-OST0000 LustreError: 24931:0:(tgt_mount.c:132:server_deregister_mount()) lustre-OST0000 not registered Lustre: server umount lustre-OST0000 complete LustreError: 24931:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -99 LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 22992:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708033877 with bad export cookie 9020486199798694523 LustreError: 22992:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.41@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 26032:0:(lib-ptl.c:1004:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.41@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_119_g0dcc848 LNet: Added LNI 192.168.123.41@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: centos-39.localnet: executing set_hostid Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 829:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708034013 with bad export cookie 15694267399548995606 LustreError: 166-1: MGC192.168.123.41@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 829:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Found index 0 for lustre-OST0000, updating log Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 7676:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708034046 with bad export cookie 15694267399548996929 LustreError: 7676:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: 166-1: MGC192.168.123.41@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 11061:0:(lib-ptl.c:1004:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.41@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_119_g0dcc848 LNet: Added LNI 192.168.123.41@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-39.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 12230:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1708034122 with bad export cookie 9067019381435477551 LustreError: 166-1: MGC192.168.123.41@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0002-lwp-OST0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88024dee02f8 IP: [<ffffffffa136e078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ed8b067 PTE 800000024dee0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw floppy i2c_core libata [last unloaded: libcfs] CPU: 10 PID: 1860 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880249f6dc40 ti: ffff88026d7e0000 task.ti: ffff88026d7e0000 RIP: 0010:[<ffffffffa136e078>] [<ffffffffa136e078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88026d7e3cc8 EFLAGS: 00010246 RAX: ffffffffa13c4820 RBX: ffff880086394548 RCX: ffff8800b4dde288 RDX: 0000000000000001 RSI: ffff88024dee02f0 RDI: ffff88026d7e3cf0 RBP: ffff88026d7e3ce0 R08: ffff88007327bfc8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000a R12: ffff88024dee0000 R13: ffff88026d7e3cf0 R14: ffffffffa13d42c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88024dee02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a2f24>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a3360>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a4f85>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ba3ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03bad28>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039ea18>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039eea2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=848 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000-mdc-ffff88028deb2548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 56 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 162 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008f15e440 x1790391257457088/t317827580263(317827580263) o101->lustre-MDT0000-mdc-ffff88028deb2548@0@lo:12/10 lens 624/608 e 0 to 0 dl 1707453496 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.103@tcp (at 0@lo) Lustre: Skipped 54 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3944 to 0x300000403:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3945 to 0x340000403:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:3977 to 0x380000405:4001 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:3949 to 0x2c0000405:3969 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3 to 0x380000403:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:3 to 0x340000404:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:4 to 0x300000404:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:4 to 0x2c0000403:65 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 11-0: lustre-MDT0002-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 13 previous similar messages LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0002-mdc-ffff88028deb2548: can't stat MDS #0: rc = -107 LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0002-mdc-ffff88028deb2548: can't stat MDS #0: rc = -19 LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0002-mdc-ffff88028deb2548: can't stat MDS #0: rc = -19 LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0002-mdc-ffff88028deb2548: can't stat MDS #0: rc = -19 LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0002-mdc-ffff88028deb2548: can't stat MDS #0: rc = -19 LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) Skipped 1 previous similar message LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88032764e440 x1790391257453440/t4294967862(4294967862) o101->lustre-MDT0002-mdc-ffff88028deb2548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1707453551 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 17 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:4 to 0x300000405:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:44 to 0x2c0000404:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:3 to 0x380000404:65 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008f15e440 x1790391257457088/t317827580263(317827580263) o101->lustre-MDT0000-mdc-ffff88028deb2548@0@lo:12/10 lens 624/608 e 0 to 0 dl 1707453576 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4110 to 0x380000405:4129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:4077 to 0x340000403:4097 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:4046 to 0x2c0000405:4065 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4067 to 0x300000403:4097 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:4 to 0x300000404:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:4 to 0x2c0000403:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3 to 0x380000403:97 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:3 to 0x340000404:97 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) lustre-MDT0002-mdc-ffff88028deb2548: can't stat MDS #0: rc = -107 LustreError: 1112:0:(lmv_obd.c:1383:lmv_statfs()) Skipped 1 previous similar message LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88032764e440 x1790391257453440/t4294967862(4294967862) o101->lustre-MDT0002-mdc-ffff88028deb2548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1707453631 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 1644:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 30 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:4 to 0x300000405:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:3 to 0x380000404:97 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:44 to 0x2c0000404:161 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 25998:0:(osd_handler.c:698:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 25998:0:(osd_handler.c:698:osd_ro()) Skipped 6 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 166-1: MGC192.168.123.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 3 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4239 to 0x300000403:4257 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4271 to 0x380000405:4289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:4180 to 0x2c0000405:4225 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:4241 to 0x340000403:4257 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:3 to 0x340000404:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:4 to 0x2c0000403:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3 to 0x380000403:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:4 to 0x300000404:129 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 9 times Lustre: Failing over lustre-MDT0002 | Link to test |
| conf-sanity test 112a: mount OST with no_create option | BUG: unable to handle kernel paging request at ffff8802b92702f8 IP: [<ffffffffa137223d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea30067 PTE 80000002b9270060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk libata i2c_core [last unloaded: libcfs] CPU: 15 PID: 3861 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88006f87a4f0 ti: ffff8802b8264000 task.ti: ffff8802b8264000 RIP: 0010:[<ffffffffa137223d>] [<ffffffffa137223d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802b8267cc8 EFLAGS: 00010246 RAX: 0000000000000000 RBX: ffff88021ce41158 RCX: ffff8802983d7f5c RDX: 000000000000000f RSI: ffff8802b92702f0 RDI: ffff88031eb88788 RBP: ffff8802b8267ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802b9270000 R13: ffff8802b8267cf0 R14: ffffffffa13d82c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802b92702f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c1f24>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c2360>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c3f85>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d93ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d9d28>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03bda18>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03bdea2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0001: new disk, initializing Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:3 to 0x2c0000402:33 Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:ost Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost] Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880325955d28.ost_server_uuid 50 LustreError: 2113:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880325955d28.ost_server_uuid in FULL state after 0 sec LustreError: 2113:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 LustreError: 2113:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff880325955d28.ost_server_uuid 50 Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff880325955d28.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 Lustre: lustre-OST0001-osc-MDT0002: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0001 complete LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 30620:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1707338371 with bad export cookie 16962418958698526315 LustreError: 30620:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 3869:0:(lib-ptl.c:1004:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.71@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_119_g0dcc848 LNet: Added LNI 192.168.123.71@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0001: deleting orphan objects from 0x300000402:3 to 0x300000402:33 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:36 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0001-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0001-osc-MDT0002: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0001 complete Lustre: Unmounted lustre-client LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: 10591:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1707338532/real 1707338532] req@ffff8802eff16940 x1790274065536064/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1707338542 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages LustreError: 5046:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1707338544 with bad export cookie 14538271088829882591 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: 4329:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1707338532/real 1707338532] req@ffff880281368f40 x1790274065535488/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1707338548 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 4320:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1707338534/real 1707338534] req@ffff88009e1d0040 x1790274065536768/t0(0) o400->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1707338550 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages Lustre: 10828:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1707338544/real 1707338544] req@ffff88021c2f8040 x1790274065539072/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1707338554 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 11397:0:(lib-ptl.c:1004:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.71@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_119_g0dcc848 LNet: Added LNI 192.168.123.71@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:36 to 0x2c0000402:97 LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000402:3 to 0x300000402:65 Lustre: lustre-OST0002: new disk, initializing Lustre: srv-lustre-OST0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000340000400-0x0000000380000400]:2:ost Lustre: cli-lustre-OST0002-super: Allocated super-sequence [0x0000000340000400-0x0000000380000400]:2:ost] Lustre: lustre-OST0002-osc-MDT0000: update sequence from 0x100020000 to 0x340000402 Lustre: lustre-OST0003: new disk, initializing Lustre: srv-lustre-OST0003: No data found on store. Initialize space: rc = -61 Lustre: Mounted lustre-client Lustre: DEBUG MARKER: Using TIMEOUT=20 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000380000400-0x00000003c0000400]:3:ost Lustre: cli-lustre-OST0003-super: Allocated super-sequence [0x0000000380000400-0x00000003c0000400]:3:ost] Lustre: Modifying parameter general.lod.*.mdt_hash in log params Lustre: lustre-OST0003-osc-MDT0000: update sequence from 0x100030000 to 0x380000402 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 13068:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1707338642 with bad export cookie 11606966942548074455 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete | Link to test |
| recovery-small test 110k: FID_QUERY failed during recovery | BUG: unable to handle kernel paging request at ffff880290de82f8 IP: [<ffffffffa136323d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb73067 PTE 8000000290de8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk i2c_core libata [last unloaded: libcfs] CPU: 14 PID: 21966 Comm: kworker/u32:5 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800a99edc40 ti: ffff8802cf744000 task.ti: ffff8802cf744000 RIP: 0010:[<ffffffffa136323d>] [<ffffffffa136323d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802cf747cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8800753f5698 RCX: ffff8802be4824e9 RDX: 000000000000000e RSI: ffff880290de82f0 RDI: ffff880326bddad8 RBP: ffff8802cf747ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880290de8000 R13: ffff8802cf747cf0 R14: ffffffffa13c92c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880290de82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0364f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0365350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0366f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa037c39e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa037cd18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0360a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa0360e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: Unmounted lustre-client Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 25 previous similar messages Lustre: *** cfs_fail_loc=1103, val=0*** Lustre: lustre-MDT0001: Aborting client recovery LustreError: 25818:0:(ldlm_lib.c:2934:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery Lustre: 25959:0:(ldlm_lib.c:2317:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: 25959:0:(ldlm_lib.c:2317:target_recovery_overseer()) Skipped 2 previous similar messages Lustre: 25959:0:(genops.c:1482:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client lustre-MDT0002-mdtlov_UUID@<unknown> Lustre: 25959:0:(genops.c:1482:class_disconnect_stale_exports()) Skipped 2 previous similar messages Lustre: lustre-MDT0001: disconnecting 2 stale clients Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] Lustre: lustre-MDT0002-osp-MDT0001: cancel update llog [0x280000402:0x1:0x0] Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:616 to 0x2c0000401:641 Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:610 to 0x380000401:641 Lustre: lustre-OST0002: deleting orphan objects from 0x340000400:610 to 0x340000400:641 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:611 to 0x300000401:641 Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) | Link to test |
| replay-single test 80b: DNE: create remote dir, drop update rep from MDT0, fail MDT1 | BUG: unable to handle kernel paging request at ffff880085b902f8 IP: [<ffffffffa1351078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f9d3067 PTE 8000000085b90060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk libata floppy serio_raw i2c_core [last unloaded: libcfs] CPU: 6 PID: 29319 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008fbd0010 ti: ffff8802ef38c000 task.ti: ffff8802ef38c000 RIP: 0010:[<ffffffffa1351078>] [<ffffffffa1351078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802ef38fcc8 EFLAGS: 00010246 RAX: ffffffffa13a7820 RBX: ffff8802f1cb5698 RCX: ffff8802acc35cc8 RDX: 0000000000000001 RSI: ffff880085b902f0 RDI: ffff8802ef38fcf0 RBP: ffff8802ef38fce0 R08: ffff8802b273a538 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000006 R12: ffff880085b90000 R13: ffff8802ef38fcf0 R14: ffffffffa13b72c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880085b902f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b8f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b9350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03baf75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d039e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d0d18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b4a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b4e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 7437:0:(ldlm_lib.c:3278:target_send_reply_msg()) @@@ dropping reply req@ffff8802c22cf840 x1789566345759872/t356482285580(0) o1000->lustre-MDT0001-mdtlov_UUID@0@lo:493/0 lens 2296/4320 e 0 to 0 dl 1706669688 ref 1 fl Interpret:/200/0 rc 0/0 job:'osp_up0-1.0' uid:0 gid:0 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0001 LustreError: 758:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x2000284a1:0x4:0x0].0x72339482 (ffff880274a751c0) refcount nonzero (1) after lock cleanup; forcing cleanup. | Link to test |
| conf-sanity test 81: sparse OST indexing | BUG: unable to handle kernel paging request at ffff8802e2c682f8 IP: [<ffffffffa138323d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e8e2067 PTE 80000002e2c68060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy libata i2c_core [last unloaded: libcfs] CPU: 5 PID: 26515 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880329589280 ti: ffff88009d8a0000 task.ti: ffff88009d8a0000 RIP: 0010:[<ffffffffa138323d>] [<ffffffffa138323d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88009d8a3cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802d7a5b3f8 RCX: ffff8802c98979a0 RDX: 0000000000000005 RSI: ffff8802e2c682f0 RDI: ffff880231550bd8 RBP: ffff88009d8a3ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802e2c68000 R13: ffff88009d8a3cf0 R14: ffffffffa13e92c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802e2c682f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0376f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0377350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0378f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa038e39e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa038ed18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0372a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa0372e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-169.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 1 previous similar message Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 3 previous similar messages Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: DEBUG MARKER: centos-169.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-169.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-169.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: Mounted lustre-client Lustre: lustre-OSTfffe-osc-MDT0000: update sequence from 0x1fffe0000 to 0x340000402 Lustre: lustre-OST68c7-osc-MDT0000: update sequence from 0x168c70000 to 0x300000402 Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 7988:0:(file.c:246:ll_close_inode_openhandle()) lustre-clilmv-ffff8802dc380958: inode [0x200000403:0x2:0x0] mdc close failed: rc = -108 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 12 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 32697:0:(ldlm_lockd.c:2674:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706337490 with bad export cookie 3232614701803031007 LustreError: 32697:0:(ldlm_lockd.c:2674:ldlm_cancel_handler()) Skipped 5 previous similar messages LustreError: 166-1: MGC192.168.123.171@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 13 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | BUG: unable to handle kernel paging request at ffff8802f7b002f8 IP: [<ffffffffa13a223d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e83b067 PTE 80000002f7b00060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm ata_piix drm_panel_orientation_quirks floppy libata virtio_blk serio_raw i2c_core [last unloaded: libcfs] CPU: 9 PID: 31928 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880275ffb760 ti: ffff880247ad8000 task.ti: ffff880247ad8000 RIP: 0010:[<ffffffffa13a223d>] [<ffffffffa13a223d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880247adbcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802eadbe7e8 RCX: ffff880252d97cdb RDX: 0000000000000009 RSI: ffff8802f7b002f0 RDI: ffff880291f2a9c8 RBP: ffff880247adbce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802f7b00000 R13: ffff880247adbcf0 R14: ffffffffa14082c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802f7b002f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03abf14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ac350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03adf75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c339e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c3d18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a7a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a7e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 17567:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880298cd2340 x1789202021597376/t154618822715(154618822715) o101->lustre-MDT0000-mdc-ffff88027a052e98@0@lo:12/10 lens 576/608 e 0 to 0 dl 1706318837 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 17567:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 9 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1671 to 0x340000403:1697 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1673 to 0x2c0000403:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1672 to 0x380000403:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1672 to 0x300000403:1697 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 15868:0:(lmv_obd.c:1340:lmv_statfs()) lustre-MDT0001-mdc-ffff880296e50008: can't stat MDS #0: rc = -107 LustreError: 15868:0:(lmv_obd.c:1340:lmv_statfs()) lustre-MDT0001-mdc-ffff880296e50008: can't stat MDS #0: rc = -19 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:246 to 0x380000404:289 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:249 to 0x340000405:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:247 to 0x2c0000405:289 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:249 to 0x300000404:289 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 17 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:27 to 0x300000405:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:27 to 0x2c0000404:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:35 to 0x380000405:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:30 to 0x340000404:65 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1715 to 0x340000403:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1714 to 0x2c0000403:1729 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1713 to 0x300000403:1729 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1715 to 0x380000403:1761 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:318 to 0x300000404:353 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:313 to 0x380000404:353 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:318 to 0x340000405:353 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:318 to 0x2c0000405:353 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times Lustre: Failing over lustre-MDT0002 LustreError: 11-0: lustre-MDT0002-mdc-ffff88027a052e98: operation ldlm_enqueue to node 0@lo failed: rc = -107 LustreError: Skipped 28 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002-lwp-OST0002: Connection restored to 192.168.123.123@tcp (at 0@lo) Lustre: Skipped 141 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:74 to 0x2c0000404:97 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:75 to 0x340000404:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:138 to 0x380000405:161 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:75 to 0x300000405:97 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 54 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1746 to 0x2c0000403:1761 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1778 to 0x340000403:1793 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1747 to 0x300000403:1793 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1778 to 0x380000403:1793 Lustre: DEBUG MARKER: centos-121.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 115 previous similar messages | Link to test |
| replay-single test 116b: large update log slave MDT recovery | BUG: unable to handle kernel paging request at ffff8802508a02f8 IP: [<ffffffffa1383078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ed76067 PTE 80000002508a0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks serio_raw libata virtio_blk i2c_core floppy [last unloaded: libcfs] CPU: 5 PID: 10300 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800a9843760 ti: ffff8802f89b4000 task.ti: ffff8802f89b4000 RIP: 0010:[<ffffffffa1383078>] [<ffffffffa1383078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802f89b7cc8 EFLAGS: 00010246 RAX: ffffffffa13d9820 RBX: ffff88021eaa5698 RCX: ffff880278a79e38 RDX: 0000000000000001 RSI: ffff8802508a02f0 RDI: ffff8802f89b7cf0 RBP: ffff8802f89b7ce0 R08: ffff8802d506b528 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000005 R12: ffff8802508a0000 R13: ffff8802f89b7cf0 R14: ffffffffa13e92c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802508a02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0310f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0311350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0312f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa032839e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa0328d18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa030ca08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa030ce92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: *** cfs_fail_loc=1702, val=0*** Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 50d: lazystatfs client/server conn race | BUG: unable to handle kernel paging request at ffff88029d6402f8 IP: [<ffffffffa136623d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb0e067 PTE 800000029d640060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy i2c_core libata virtio_blk [last unloaded: libcfs] CPU: 1 PID: 23291 Comm: kworker/u32:5 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031e569280 ti: ffff8802704a0000 task.ti: ffff8802704a0000 RIP: 0010:[<ffffffffa136623d>] [<ffffffffa136623d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802704a3cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88028ab9a2a8 RCX: ffff8802b21c65e3 RDX: 0000000000000001 RSI: ffff88029d6402f0 RDI: ffff8803276b2f18 RBP: ffff8802704a3ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88029d640000 R13: ffff8802704a3cf0 R14: ffffffffa13cc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88029d6402f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b5f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b6350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b7f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03cd39e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cdd18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b1a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b1e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0001: new disk, initializing Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:ost Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0001 complete Lustre: server umount lustre-MDT0000 complete LustreError: 22980:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706285860 with bad export cookie 17538675416044165341 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: centos-74.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 4 previous similar messages Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: Skipped 2 previous similar messages Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: Skipped 2 previous similar messages Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 21 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 49 previous similar messages LustreError: 1673:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706286064 with bad export cookie 17538675416044167091 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test |
| conf-sanity test 112a: mount OST with no_create option | BUG: unable to handle kernel paging request at ffff8800a16382f8 IP: [<ffffffffa135923d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f8f5067 PTE 80000000a1638060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw libata i2c_core virtio_blk [last unloaded: libcfs] CPU: 7 PID: 2728 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802de1224f0 ti: ffff88027fd0c000 task.ti: ffff88027fd0c000 RIP: 0010:[<ffffffffa135923d>] [<ffffffffa135923d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88027fd0fcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88009ea31158 RCX: ffff8802720d0125 RDX: 0000000000000007 RSI: ffff8800a16382f0 RDI: ffff880255a4bb68 RBP: ffff88027fd0fce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8800a1638000 R13: ffff88027fd0fcf0 R14: ffffffffa13bf2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800a16382f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa039df14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa039e350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa039ff75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b539e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b5d18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0399a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa0399e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (not set up) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:3 to 0x2c0000402:65 Lustre: lustre-OST0001: new disk, initializing Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:ost Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost] LustreError: 26638:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0002: cannot cleanup orphans: rc = -1 LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 Lustre: Mounted lustre-client LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) Skipped 2 previous similar messages LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) Skipped 5 previous similar messages Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880297e80008.ost_server_uuid 50 LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) lustre-OST0001-osc-MDT0000: cannot cleanup orphans: rc = -1 LustreError: 26699:0:(osp_precreate.c:986:osp_precreate_cleanup_orphans()) Skipped 11 previous similar messages Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880297e80008.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff880297e80008.ost_server_uuid 50 Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff880297e80008.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 LustreError: 11-0: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0001 complete Lustre: Unmounted lustre-client LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 22991:0:(ldlm_lockd.c:2674:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706214555 with bad export cookie 5818227326843107890 LustreError: 22991:0:(ldlm_lockd.c:2674:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.216@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 28674:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.216@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_75_g7189ec1 LNet: Added LNI 192.168.123.216@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:68 to 0x2c0000402:97 Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0001: deleting orphan objects from 0x300000402:3 to 0x300000402:33 Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-214.localnet: executing wait_import_state (FULL|IDLE) os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0001-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0001-osc-MDT0002: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0001 complete Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages LustreError: 29780:0:(ldlm_lockd.c:2674:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706214812 with bad export cookie 7324813937293043585 LustreError: 166-1: MGC192.168.123.216@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) | Link to test |
| conf-sanity test 50f: normal statfs one server in down | BUG: unable to handle kernel paging request at ffff8803249a02f8 IP: [<ffffffffa136e23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 23c4067 PMD 33febd067 PTE 80000003249a0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks libata virtio_blk serio_raw i2c_core floppy [last unloaded: libcfs] CPU: 15 PID: 3691 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800b5dd1280 ti: ffff8802f0800000 task.ti: ffff8802f0800000 RIP: 0010:[<ffffffffa136e23d>] [<ffffffffa136e23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802f0803cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802986a33f8 RCX: ffff880289a360af RDX: 000000000000000f RSI: ffff8803249a02f0 RDI: ffff8802dd013f68 RBP: ffff8802f0803ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8803249a0000 R13: ffff8802f0803cf0 R14: ffffffffa13d42c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8803249a02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa038bf14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa038c350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa038df75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a339e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03a3d18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0387a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa0387e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0001: new disk, initializing Lustre: Skipped 3 previous similar messages Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 4 previous similar messages Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:ost Lustre: Skipped 3 previous similar messages Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost] Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in DISCONN state after 0 sec Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid LustreError: 167-0: lustre-OST0001-osc-MDT0002: This client was evicted by lustre-OST0001; in progress operations using this service will fail. Lustre: lustre-OST0001-osc-MDT0002: Connection restored to 192.168.123.71@tcp (at 0@lo) Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec Lustre: server umount lustre-OST0001 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 24954:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706107778 with bad export cookie 14711039834764106086 LustreError: 24954:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test |
| conf-sanity test 38: MDS recreates missing lov_objid file from OST data | BUG: unable to handle kernel paging request at ffff8802860802f8 IP: [<ffffffffa138723d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ebc9067 PTE 8000000286080060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon pcspkr virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs] CPU: 9 PID: 29320 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802875024f0 ti: ffff8800b3da8000 task.ti: ffff8800b3da8000 RIP: 0010:[<ffffffffa138723d>] [<ffffffffa138723d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8800b3dabcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802d9bc33f8 RCX: ffff880329189b8c RDX: 0000000000000009 RSI: ffff8802860802f0 RDI: ffff8802c0918dd8 RBP: ffff8800b3dabce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880286080000 R13: ffff8800b3dabcf0 R14: ffffffffa13ed2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802860802f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b0f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b1350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b2f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c839e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c8d18>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03aca08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03ace92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages LustreError: 18789:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706075436 with bad export cookie 3151042318403124144 LustreError: 18789:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.86@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation out_update to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 20111:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0002: osp_attr_get update error [0x240000bd2:0x1:0x0]: rc = -5 LustreError: 20111:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880318a85f40 x1788949539406464/t0(0) o1000->lustre-MDT0001-osp-MDT0002@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 20111:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -5 LustreError: 20111:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -5 Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: delete lov_objid file on MDS Lustre: Evicted from MGS (at 192.168.123.86@tcp) after server handle changed from 0x0 to 0x2bbac04b2ca4472c Lustre: MGC192.168.123.86@tcp: Connection restored to 192.168.123.86@tcp (at 0@lo) Lustre: lustre-MDT0002-lwp-OST0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: 6983:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1706075446/real 1706075446] req@ffff880318a86940 x1788949539406784/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1706075462 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000bd0 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 21 previous similar messages Lustre: 6980:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1706075451/real 1706075451] req@ffff880318a82340 x1788949539406848/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1706075467 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:65 Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.123.86@tcp (at 0@lo) Lustre: 6981:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1706075456/real 1706075456] req@ffff880318a85f40 x1788949539406912/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1706075472 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:65 Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: 6980:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1706075461/real 1706075461] req@ffff880318a80f40 x1788949539406976/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1706075477 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-MDT0002-lwp-OST0000: Connection restored to (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 18777:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706075492 with bad export cookie 3151042318403127084 LustreError: 166-1: MGC192.168.123.86@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 18777:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 8 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete LustreError: 6973:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802be529440 x1788949539439552/t0(0) o250->MGC192.168.123.86@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: Evicted from MGS (at 192.168.123.86@tcp) after server handle changed from 0x0 to 0x2bbac04b2ca45143 Lustre: MGC192.168.123.86@tcp: Connection restored to 192.168.123.86@tcp (at 0@lo) Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000bd0:3 to 0x2c0000bd0:33 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:97 Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.123.86@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:97 Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-84.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Mounted lustre-client Lustre: lustre-MDT0002-lwp-OST0000: Connection restored to 192.168.123.86@tcp (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 22 previous similar messages LustreError: 18793:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1706075548 with bad export cookie 3151042318403129667 LustreError: 166-1: MGC192.168.123.86@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 18793:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages | Link to test |
| conf-sanity test 68: be able to reserve specific sequences in FLDB | BUG: unable to handle kernel paging request at ffff88006e3802f8 IP: [<ffffffffa135e23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fc02067 PMD 33fa90067 PTE 800000006e380060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks i2c_core libata serio_raw virtio_blk floppy [last unloaded: libcfs] CPU: 2 PID: 30735 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008359b760 ti: ffff88027e9f4000 task.ti: ffff88027e9f4000 RIP: 0010:[<ffffffffa135e23d>] [<ffffffffa135e23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88027e9f7cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880270cd22a8 RCX: ffff8802a4a88661 RDX: 0000000000000002 RSI: ffff88006e3802f0 RDI: ffff880096164b88 RBP: ffff88027e9f7ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88006e380000 R13: ffff88027e9f7cf0 R14: ffffffffa13c42c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88006e3802f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0392f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0393350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0394f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03aa56e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03aaee8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa038ea08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa038ee92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: ctl-lustre-MDT0000: [0x0000000300000400-0x0000000340000400]:0:mdt sequences allocated: rc = 0 Lustre: Mounted lustre-client Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000340000400-0x0000000380000400]:0:mdt Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 27611:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705854800 with bad export cookie 15056550024312652077 LustreError: 27611:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 31614:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.46@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:65 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 593:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705854862 with bad export cookie 7329809294946890697 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 4348:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.46@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 5432:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.46@tcp: failed processing log, type 1: rc = -5 LustreError: 5432:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.46@tcp: failed processing log, type 4: rc = -110 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:97 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 5464:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705855022 with bad export cookie 17066170876440111146 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 9560:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.46@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:129 Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 13852:0:(file.c:5393:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -108 LustreError: 10696:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705855093 with bad export cookie 3601225120038370691 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 14742:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.46@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:161 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 15857:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705855174 with bad export cookie 9700946805443870592 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 19947:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.46@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:2:mdt] Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0002.mdt_server_uuid 50 Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0002.mdt_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0002.mdt_server_uuid 50 Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0002.mdt_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0002-osp-MDT0000.mdt_server_uuid 50 Lustre: DEBUG MARKER: osp.lustre-MDT0002-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0002-osp-MDT0001.mdt_server_uuid 50 Lustre: DEBUG MARKER: osp.lustre-MDT0002-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0002-mdc-ffff88024fad6678: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete Key type lgssc unregistered LNet: 4126:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.46@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.46@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Found index 0 for lustre-OST0000, updating log Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-44.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages Lustre: 9185:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705855382/real 1705855382] req@ffff8802eb4220c0 x1788718970070720/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1705855392 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 6339:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705855394 with bad export cookie 17404394524933830852 LustreError: 166-1: MGC192.168.123.46@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: Skipped 1 previous similar message Lustre: 5609:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705855381/real 1705855381] req@ffff8802b2138040 x1788718970069888/t0(0) o400->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1705855397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 5611:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705855382/real 1705855382] req@ffff8802eb4213c0 x1788718970070784/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1705855398 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: 5607:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705855386/real 1705855386] req@ffff8802b213e1c0 x1788718970071552/t0(0) o400->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1705855402 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages | Link to test |
| conf-sanity test 22: start a client before osts (should return errs) | BUG: unable to handle kernel paging request at ffff880325e682f8 IP: [<ffffffffa133d23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 23c4067 PMD 33feb2067 PTE 8000000325e68060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw floppy i2c_core virtio_blk libata [last unloaded: libcfs] CPU: 11 PID: 19094 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802dd3c5c40 ti: ffff880290a94000 task.ti: ffff880290a94000 RIP: 0010:[<ffffffffa133d23d>] [<ffffffffa133d23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880290a97cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880283cf9158 RCX: ffff8802c68c4381 RDX: 000000000000000b RSI: ffff880325e682f0 RDI: ffff880263c09828 RBP: ffff880290a97ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880325e68000 R13: ffff880290a97cf0 R14: ffffffffa13a32c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880325e682f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0381f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0382350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0383f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa039956e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa0399ee8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa037da08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa037de92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 10 sec Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete Lustre: Mounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0002: Connection restored to 192.168.123.106@tcp (at 0@lo) LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.123.106@tcp (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880231c49bf8.ost_server_uuid 50 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880231c49bf8.ost_server_uuid in FULL state after 0 sec LustreError: 29739:0:(ldlm_lockd.c:779:ldlm_handle_ast_error()) ### client (nid 0@lo) returned error from blocking AST (req@ffff880290528d40 x1788628039820672 status -107 rc -107), evict it ns: mdt-lustre-MDT0000_UUID lock: ffff88009b5a8040/0xc36a9d50a1c62791 lrc: 4/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xc36a9d50a1c62783 expref: 5 pid: 29731 timeout: 86536 lvb_type: 0 LustreError: 138-a: lustre-MDT0000: A client on nid 0@lo was evicted due to a lock blocking callback time out: rc -107 LustreError: 29720:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 0@lo ns: mdt-lustre-MDT0000_UUID lock: ffff88009b5a8040/0xc36a9d50a1c62791 lrc: 3/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x60200400000020 nid: 0@lo remote: 0xc36a9d50a1c62783 expref: 6 pid: 29731 timeout: 0 lvb_type: 0 Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 11 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 29707:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705769308 with bad export cookie 14081240154564338313 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 29 previous similar messages Lustre: 20100:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705769301/real 1705769301] req@ffff8802bb9413c0 x1788628039824704/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1705769317 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 20100:0:(client.c:2337:ptlrpc_expire_one_request()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 3096:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.106@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.106@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 9 previous similar messages LustreError: 4201:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705769408 with bad export cookie 18389354585392011747 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: Unmounted lustre-client LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 18 previous similar messages LustreError: 3507:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff88022ebe5b40 x1788628801974336/t0(0) o250->MGC192.168.123.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: lustre-MDT0000: Denying connection for new client 357f7ee0-54b0-48d2-bce4-10c2701e1344 (at 0@lo), waiting for 3 known clients (0 recovered, 0 in progress, and 0 evicted) to recover in 0:59 Lustre: lustre-MDT0000-lwp-MDT0002: Connection restored to 192.168.123.106@tcp (at 0@lo) LustreError: 11-0: lustre-MDT0000-mdc-ffff880093aa6678: operation mds_connect to node 0@lo failed: rc = -16 Lustre: lustre-MDT0000: Denying connection for new client 357f7ee0-54b0-48d2-bce4-10c2701e1344 (at 0@lo), waiting for 3 known clients (2 recovered, 0 in progress, and 0 evicted) to recover in 1:04 Lustre: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0000-mdc-ffff880093aa6678: operation mds_connect to node 0@lo failed: rc = -16 LustreError: Skipped 1 previous similar message LustreError: 7777:0:(lmv_obd.c:1340:lmv_statfs()) lustre-MDT0000-mdc-ffff880093aa6678: can't stat MDS #0: rc = -16 Lustre: Unmounted lustre-client LustreError: 7777:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -16 Lustre: lustre-MDT0002: haven't heard from client 5babf4b0-7d47-40fd-9885-1feb80d2e6c1 (at 0@lo) in 31 seconds. I think it's dead, and I am evicting it. exp ffff8802cff92548, cur 1705769427 expire 1705769397 last 1705769396 LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 8071:0:(ldlm_lib.c:2934:target_stop_recovery_thread()) lustre-MDT0000: Aborting recovery Lustre: 7712:0:(ldlm_lib.c:2317:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] LustreError: 7712:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802af775b40 x1788628801987008/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 7712:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-lustre-MDT0001-osp-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 7712:0:(fid_request.c:335:seq_client_alloc_fid()) cli-cli-lustre-MDT0001-osp-MDT0000: Can't allocate new sequence: rc = -5 Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x280000401:0x1:0x0] Lustre: lustre-MDT0000: Recovery over after 0:15, of 3 clients 0 recovered and 3 were evicted. Lustre: 7712:0:(mdt_handler.c:7831:mdt_postrecov()) lustre-MDT0000: auto trigger paused LFSCK failed: rc = -6 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: 4201:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705769437 with bad export cookie 18389354585392013182 LustreError: 3507:0:(import.c:674:ptlrpc_connect_import_locked()) can't connect to a closed import LustreError: 4201:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages LustreError: 3507:0:(import.c:674:ptlrpc_connect_import_locked()) can't connect to a closed import Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 8844:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.106@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.106@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:97 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 13362:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705769509 with bad export cookie 7482529072258102622 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test |
| replay-single test 136: MDS to disconnect all OSPs first, then cleanup ldlm | BUG: unable to handle kernel paging request at ffff8802883682f8 IP: [<ffffffffa1369078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ebb8067 PTE 8000000288368060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix virtio_blk serio_raw floppy libata i2c_core [last unloaded: libcfs] CPU: 0 PID: 26720 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88024aa649d0 ti: ffff880073828000 task.ti: ffff880073828000 RIP: 0010:[<ffffffffa1369078>] [<ffffffffa1369078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88007382bcc8 EFLAGS: 00010246 RAX: ffffffffa13bf820 RBX: ffff880290295698 RCX: ffff8802d827de38 RDX: 0000000000000001 RSI: ffff8802883682f0 RDI: ffff88007382bcf0 RBP: ffff88007382bce0 R08: ffff880287bf1548 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000000 R12: ffff880288368000 R13: ffff88007382bcf0 R14: ffffffffa13cf2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802883682f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa03d5f14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03d6350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03d7f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ed56e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03edee8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03d1a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03d1e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: *** cfs_fail_loc=170b, val=0*** Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 38: MDS recreates missing lov_objid file from OST data | BUG: unable to handle kernel paging request at ffff8802b2e602f8 IP: [<ffffffffa136f078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea62067 PTE 80000002b2e60060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw libata i2c_core [last unloaded: libcfs] CPU: 15 PID: 23965 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802e8ccb760 ti: ffff8802830b8000 task.ti: ffff8802830b8000 RIP: 0010:[<ffffffffa136f078>] [<ffffffffa136f078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802830bbcc8 EFLAGS: 00010246 RAX: ffffffffa13c5820 RBX: ffff880297b1c548 RCX: ffff8802c01011a2 RDX: 0000000000000001 RSI: ffff8802b2e602f0 RDI: ffff8802830bbcf0 RBP: ffff8802830bbce0 R08: 5f79656b5f646f6c R09: 74696e695f79656b R10: 0000000000000000 R11: 000000000000000f R12: ffff8802b2e60000 R13: ffff8802830bbcf0 R14: ffffffffa13d52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802b2e602f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03adf14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ae350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03aff75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c556e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c5ee8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a9a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a9e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity Lustre: Unmounted lustre-client Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 32640:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705682654 with bad export cookie 5373112677409696055 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 32640:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 9 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: delete lov_objid file on MDS LustreError: 20636:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8800992c86c0 x1788537755212992/t0(0) o250->MGC192.168.123.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000bd0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:65 Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.123.106@tcp (at 0@lo) Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:65 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Mounted lustre-client Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.123.106@tcp (at 0@lo) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 32640:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705682702 with bad export cookie 5373112677409698995 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 32640:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 9 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete LustreError: 20636:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802f0ca13c0 x1788537755244032/t0(0) o250->MGC192.168.123.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000bd0:3 to 0x2c0000bd0:33 Lustre: lustre-MDT0002-lwp-OST0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:97 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.123.106@tcp (at 0@lo) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Mounted lustre-client Lustre: 20646:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705682707/real 1705682707] req@ffff8802f0ca2dc0 x1788537755243968/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1705682723 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages Lustre: 20643:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1705682712/real 1705682712] req@ffff8802f0ca2740 x1788537755244096/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1705682728 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 7115:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705682732 with bad export cookie 5373112677409701550 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 7115:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: files compared the same Lustre: server umount lustre-OST0000 complete Key type lgssc unregistered LNet: 13213:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.106@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_26_g0114a07 LNet: Added LNI 192.168.123.106@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:41 to 0x2c0000401:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000bd0:35 to 0x2c0000bd0:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 14356:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705682814 with bad export cookie 7004852112916913872 LustreError: 166-1: MGC192.168.123.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete | Link to test |
| replay-dual test 22a: c1 lfs mkdir -i 1 dir1, M1 drop reply | BUG: unable to handle kernel paging request at ffff8802391f02f8 IP: [<ffffffffa135c23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33effc067 PMD 33ee33067 PTE 80000002391f0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata virtio_blk floppy serio_raw i2c_core [last unloaded: libcfs] CPU: 15 PID: 30408 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88028453b760 ti: ffff880284450000 task.ti: ffff880284450000 RIP: 0010:[<ffffffffa135c23d>] [<ffffffffa135c23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880284453cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802c03233f8 RCX: ffff880290afb19f RDX: 000000000000000f RSI: ffff8802391f02f0 RDI: ffff8802a6fbf748 RBP: ffff880284453ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802391f0000 R13: ffff880284453cf0 R14: ffffffffa13c22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802391f02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03bcf14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03bd350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03bef75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d456e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d4ee8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b8a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b8e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: *** cfs_fail_loc=119, val=2147483648*** LustreError: 10858:0:(ldlm_lib.c:3278:target_send_reply_msg()) @@@ dropping reply req@ffff8802ec4ff540 x1788486265549248/t4294967759(0) o36->b03e9844-7b03-4e12-863f-fc21268a0adc@0@lo:24/0 lens 560/448 e 0 to 0 dl 1705634869 ref 1 fl Interpret:/200/0 rc 0/0 job:'lfs.0' uid:0 gid:0 Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff8802633a82f8 IP: [<ffffffffa134623d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ece1067 PTE 80000002633a8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw virtio_blk i2c_core floppy [last unloaded: libcfs] CPU: 13 PID: 3669 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880324329280 ti: ffff8802891cc000 task.ti: ffff8802891cc000 RIP: 0010:[<ffffffffa134623d>] [<ffffffffa134623d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802891cfcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880286a5d698 RCX: ffff8802eafad03a RDX: 000000000000000d RSI: ffff8802633a82f0 RDI: ffff88009eb50688 RBP: ffff8802891cfce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802633a8000 R13: ffff8802891cfcf0 R14: ffffffffa13ac2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802633a82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03fff14>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0400350>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0401f75>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa041756e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa0417ee8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03fba08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03fbe92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:35 to 0x240000400:97 Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 21803:0:(client.c:1511:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802a6e0adc0 x1788435129976448/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1705584784 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client LustreError: 21803:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 21803:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 21803:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff88027ac882f8 IP: [<ffffffffa136123d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ec24067 PTE 800000027ac88060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic pata_acpi drm ata_piix drm_panel_orientation_quirks libata virtio_blk serio_raw i2c_core floppy [last unloaded: libcfs] CPU: 8 PID: 16022 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802832dc9d0 ti: ffff8802d7194000 task.ti: ffff8802d7194000 RIP: 0010:[<ffffffffa136123d>] [<ffffffffa136123d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802d7197cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88009953a2a8 RCX: ffff8803265f6ce5 RDX: 0000000000000008 RSI: ffff88027ac882f0 RDI: ffff880293138408 RBP: ffff8802d7197ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88027ac88000 R13: ffff8802d7197cf0 R14: ffffffffa13c72c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88027ac882f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0373f24>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0374360>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0375f85>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa038b57e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa038bef8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa036fa08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa036fe92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 21294:0:(client.c:1511:after_reply()) @@@ resending request on EINPROGRESS req@ffff8802daeca0c0 x1788290958163456/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1705447338 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client LustreError: 21294:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 21294:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 21294:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount fs15246-MDT0000 complete Lustre: server umount fs15246-OST0000 complete Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 22428:0:(import.c:355:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) LustreError: 22428:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8802991f06c0 x1788290958174656/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1705447347 ref 2 fl UnregRPC:ENU/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 LustreError: 22428:0:(import.c:389:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. LustreError: 22428:0:(import.c:355:ptlrpc_invalidate_import()) lustre-OST0000_UUID: timeout waiting for callback (1 != 0) LustreError: 22428:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff8802991f06c0 x1788290958174656/t0(0) o8->lustre-OST0000-osc-MDT0000@1.2.3.4@tcp:28/4 lens 520/544 e 0 to 0 dl 1705447347 ref 2 fl UnregRPC:ENU/200/ffffffff rc -5/-1 job:'' uid:0 gid:0 LustreError: 22428:0:(import.c:389:ptlrpc_invalidate_import()) lustre-OST0000_UUID: Unregistering RPCs found (1). Network is sluggish? Waiting for them to error out. Lustre: server umount lustre-MDT0000 complete LustreError: 17234:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705447389 with bad export cookie 5477532665837705309 LustreError: 17234:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.31@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 44 previous similar messages LustreError: Skipped 12 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 23273:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.31@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_27_g86a7c1e LNet: Added LNI 192.168.123.31@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: centos-29.localnet: executing set_hostid Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-29.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 32436:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705447594 with bad export cookie 6161792707869152033 LustreError: 166-1: MGC192.168.123.31@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 32436:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages | Link to test |
| conf-sanity test 57a: initial registration from failnode should fail (should return errs) | BUG: unable to handle kernel paging request at ffff8800804302f8 IP: [<ffffffffa136d23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f9fe067 PTE 8000000080430060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw libata i2c_core [last unloaded: libcfs] CPU: 2 PID: 7686 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008a60a4f0 ti: ffff8802cd534000 task.ti: ffff8802cd534000 RIP: 0010:[<ffffffffa136d23d>] [<ffffffffa136d23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802cd537cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802ae9922a8 RCX: ffff8800a05e2942 RDX: 0000000000000002 RSI: ffff8800804302f0 RDI: ffff8802c7bc38e8 RBP: ffff8802cd537ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880080430000 R13: ffff8802cd537cf0 R14: ffffffffa13d32c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800804302f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa039bf24>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa039c360>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa039df85>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b357e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b3ef8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0397a08>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa0397e92>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-69.localnet: executing load_modules_local Lustre: DEBUG MARKER: centos-69.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-MDT0000 complete LustreError: 31592:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384475 with bad export cookie 18137431293838809443 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Denying initial registration attempt from nid 192.168.123.71@tcp, specified as failover LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. LustreError: 8133:0:(tgt_mount.c:2216:server_fill_super()) Unable to start targets: -99 LustreError: 8133:0:(tgt_mount.c:1752:server_put_super()) no obd lustre-OST0000 LustreError: 8133:0:(tgt_mount.c:132:server_deregister_mount()) lustre-OST0000 not registered Lustre: server umount lustre-OST0000 complete LustreError: 8133:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -99 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 6226:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384506 with bad export cookie 18137431293838810710 LustreError: 6226:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 9290:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.71@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_27_g86a7c1e LNet: Added LNI 192.168.123.71@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: DEBUG MARKER: centos-69.localnet: executing set_hostid Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 16325:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384631 with bad export cookie 17542724983561101962 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 16325:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: Found index 0 for lustre-OST0000, updating log Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 22793:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384664 with bad export cookie 17542724983561103229 LustreError: 22793:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 22804:0:(mgc_request.c:619:do_requeue()) failed processing log: -5 LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 25990:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.71@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_27_g86a7c1e LNet: Added LNI 192.168.123.71@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-MDT0000 complete LustreError: 29474:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384747 with bad export cookie 4353114147054414241 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: Skipped 4 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0002-lwp-OST0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete LustreError: 26438:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802c12af540 x1788225464108160/t0(0) o250->MGC192.168.123.71@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:134 to 0x2c0000402:161 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-MDT0002-lwp-OST0000: Connection restored to 192.168.123.71@tcp (at 0@lo) Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 192.168.123.71@tcp (at 0@lo) LustreError: 167-0: lustre-MDT0000-mdc-ffff8802c8fee678: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 27160:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384804 with bad export cookie 4353114147054422529 LustreError: 27160:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 2600:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.71@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.60_27_g86a7c1e LNet: Added LNI 192.168.123.71@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 3981:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 LustreError: 3981:0:(lod_sub_object.c:932:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 LustreError: 3988:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0002-osp-MDT0000: get update log duration 8, retries 0, failed: rc = -5 LustreError: 3981:0:(lod_sub_object.c:932:lod_sub_prep_llog()) Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 3787:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384856 with bad export cookie 8600732285422592746 LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 4647:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802a92deec0 x1788225602463872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 4647:0:(lod_sub_object.c:956:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't open llog [0x280000401:0x1:0x0]: rc = -5 LustreError: 4647:0:(lod_dev.c:525:lod_sub_recovery_thread()) lustre-MDT0002-osp-MDT0001: get update log duration 14, retries 0, failed: rc = -5 LustreError: 4647:0:(lod_dev.c:525:lod_sub_recovery_thread()) Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: MGS: Regenerating lustre-MDT0002 log by user request: rc = 0 Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 13b-9: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. LustreError: 160-7: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. LustreError: 9172:0:(tgt_mount.c:2216:server_fill_super()) Unable to start targets: -2 LustreError: 9172:0:(tgt_mount.c:1752:server_put_super()) no obd lustre-OST0000 LustreError: 9172:0:(tgt_mount.c:132:server_deregister_mount()) lustre-OST0000 not registered Lustre: server umount lustre-OST0000 complete LustreError: 9172:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -2 Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:134 to 0x2c0000402:193 Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 Lustre: lustre-OST0001: new disk, initializing Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: server umount lustre-OST0001 complete Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 6852:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1705384903 with bad export cookie 8600732285422593579 LustreError: 6852:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages | Link to test |
| replay-single test 111g: DNE: unlink striped dir, fail MDT1/MDT2 | BUG: unable to handle kernel paging request at ffff8802c30302f8 IP: [<ffffffffa1341078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e9e0067 PTE 80000002c3030060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) crc32_generic lnet(OE) libcfs(OE) crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks i2c_core serio_raw virtio_blk libata floppy CPU: 8 PID: 7933 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88029299a4f0 ti: ffff880294020000 task.ti: ffff880294020000 RIP: 0010:[<ffffffffa1341078>] [<ffffffffa1341078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880294023cc8 EFLAGS: 00010246 RAX: ffffffffa1397820 RBX: ffff8802767c22a8 RCX: ffff88031dda3c68 RDX: 0000000000000001 RSI: ffff8802c30302f0 RDI: ffff880294023cf0 RBP: ffff880294023ce0 R08: ffff8802be57aa88 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000008 R12: ffff8802c3030000 R13: ffff880294023cf0 R14: ffffffffa13a72c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802c30302f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03aedc4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03af200>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b0e25>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c634e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c6cc8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03aa8b8>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03aad42>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 81c: DNE: unlink remote dir, drop MDT0 update reply, fail MDT0,MDT1 | BUG: unable to handle kernel paging request at ffff8802d2b202f8 IP: [<ffffffffa1357078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e963067 PTE 80000002d2b20060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix libata serio_raw virtio_blk i2c_core floppy [last unloaded: libcfs] CPU: 2 PID: 7490 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88027746c9d0 ti: ffff880273eb4000 task.ti: ffff880273eb4000 RIP: 0010:[<ffffffffa1357078>] [<ffffffffa1357078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880273eb7cc8 EFLAGS: 00010246 RAX: ffffffffa13ad820 RBX: ffff880273981158 RCX: ffff88031d53a568 RDX: 0000000000000001 RSI: ffff8802d2b202f0 RDI: ffff880273eb7cf0 RBP: ffff880273eb7ce0 R08: ffff88027cfa3fc8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000002 R12: ffff8802d2b20000 R13: ffff880273eb7cf0 R14: ffffffffa13bd2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802d2b202f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a0dc4>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a1200>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a2e25>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b834e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b8cc8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039c8b8>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039cd42>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:4739 to 0x380000403:5089 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:5287 to 0x2c0000403:5697 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4931 to 0x300000403:5185 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:6137 to 0x340000403:6561 Lustre: DEBUG MARKER: centos-96.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff8802128e02f8 IP: [<ffffffffa136f23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33effc067 PMD 33ef67067 PTE 80000002128e0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw floppy libata virtio_blk i2c_core [last unloaded: libcfs] CPU: 11 PID: 7687 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88022f5bc9d0 ti: ffff880324a18000 task.ti: ffff880324a18000 RIP: 0010:[<ffffffffa136f23d>] [<ffffffffa136f23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880324a1bcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802547bb3f8 RCX: ffff88007394d847 RDX: 000000000000000b RSI: ffff8802128e02f0 RDI: ffff880248e1f548 RBP: ffff880324a1bce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802128e0000 R13: ffff880324a1bcf0 R14: ffffffffa13d52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802128e02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a8334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a8770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03aa395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03bf8be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c0238>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a3e38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a42c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-104.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:35 to 0x240000400:97 Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 31763:0:(client.c:1511:after_reply()) @@@ resending request on EINPROGRESS req@ffff880097156840 x1787126792340800/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1704337053 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client LustreError: 31763:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 31763:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 31763:0:(osp_precreate.c:481:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 | Link to test |
| conf-sanity test 30a: Big config llog and permanent parameter deletion | BUG: unable to handle kernel paging request at ffff8800ad0882f8 IP: [<ffffffffa136f23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f898067 PTE 80000000ad088060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm serio_raw ata_piix drm_panel_orientation_quirks virtio_blk libata floppy i2c_core [last unloaded: libcfs] CPU: 9 PID: 20800 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800942b3760 ti: ffff8802fa974000 task.ti: ffff8802fa974000 RIP: 0010:[<ffffffffa136f23d>] [<ffffffffa136f23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802fa977cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88031f93e7e8 RCX: ffff880095889a87 RDX: 0000000000000009 RSI: ffff8800ad0882f0 RDI: ffff8802d0e80dd8 RBP: ffff8802fa977ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8800ad088000 R13: ffff8802fa977cf0 R14: ffffffffa13d52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800ad0882f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a1334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a1770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a3395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b88be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b9238>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039ce38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039d2c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Setting parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Skipped 1 previous similar message Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Skipped 1 previous similar message Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Skipped 2 previous similar messages Lustre: Modifying parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Skipped 4 previous similar messages Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: Disabling parameter lustre-client.llite.max_read_ahead_whole_mb in log lustre-client Lustre: Skipped 8 previous similar messages Lustre: Unmounted lustre-client Lustre: Mounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 3701:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704196545 with bad export cookie 11932435343939729866 LustreError: 3701:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 8 previous similar messages LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 6673:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 7845:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704196609 with bad export cookie 5619809109200946426 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 11903:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: test-33a-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter test-33a-MDT0000.mdt.identity_upcall in log test-33a-MDT0000 Lustre: ctl-test-33a-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: test-33a-MDT0000: new disk, initializing Lustre: test-33a-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-test-33a-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: test-33a-OST1fff: new disk, initializing Lustre: srv-test-33a-OST1fff: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: test-33a-OST1fff: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-test-33a-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1fff:ost Lustre: cli-test-33a-OST1fff-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1fff:ost] Lustre: test-33a-OST1fff-osc-MDT0000: update sequence from 0x11fff0000 to 0x240000400 Lustre: Mounted test-33a-client Lustre: Unmounted test-33a-client LustreError: 11-0: test-33a-OST1fff-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: test-33a-OST1fff-osc-MDT0000: Connection to test-33a-OST1fff (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: test-33a-OST1fff: Not available for connect from 0@lo (stopping) Lustre: server umount test-33a-OST1fff complete Lustre: server umount test-33a-MDT0000 complete Key type lgssc unregistered LNet: 18025:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=304, val=0*** Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 19243:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704196792 with bad export cookie 16869867022803508379 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 19243:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 23285:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: lustre-OST07c6: new disk, initializing Lustre: srv-lustre-OST07c6: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST07c6 complete Lustre: server umount lustre-MDT0000 complete Key type lgssc unregistered LNet: 29352:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:99 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 30448:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704196979 with bad export cookie 6862482157378587039 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 2137:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:131 to 0x2c0000402:161 Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-MDT0000 complete LustreError: 3313:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704197056 with bad export cookie 6439552174841009822 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: 7028:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1704197078/real 1704197078] req@ffff880323fe61c0 x1786980080840640/t0(0) o39->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1704197088 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-OST0000 complete Key type lgssc unregistered LNet: 7546:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:163 to 0x2c0000402:193 Lustre: Mounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-ffff8802d84eb7e8: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 8665:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704197163 with bad export cookie 1652310498794640189 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_disconnect to node 0@lo failed: rc = -107 Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 12898:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.51@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_110_g77f7998 LNet: Added LNI 192.168.123.51@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-49.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:225 Lustre: DEBUG MARKER: Set up a fake failnode for the MDS Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) LNetError: 120-3: Refusing connection from 192.168.123.51 for 127.0.0.2@tcp: No matching NI LNetError: 13285:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2024-01-02 7h07m16s Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LNetError: 120-3: Refusing connection from 192.168.123.51 for 127.0.0.2@tcp: No matching NI LNetError: 13286:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? Lustre: server umount lustre-MDT0000 complete Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LNetError: 120-3: Refusing connection from 192.168.123.51 for 127.0.0.2@tcp: No matching NI LNetError: 13287:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? LustreError: Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:257 Lustre: DEBUG MARKER: Wait for df (17322) ... Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to (at 0@lo) LustreError: 167-0: lustre-MDT0000-mdc-ffff8800953cae98: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: 17322:0:(statahead.c:1619:is_first_dirent()) lustre: reading dir [0x200000007:0x1:0x0] at 0 opendir_pid = 17322 : rc = -5 Lustre: DEBUG MARKER: done Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LNetError: 120-3: Refusing connection from 192.168.123.51 for 127.0.0.2@tcp: No matching NI LNetError: 13288:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? Lustre: 18369:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1704197257/real 1704197257] req@ffff8802b3cc0040 x1786980266451200/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1704197267 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 14007:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1704197270 with bad export cookie 10823616108669711705 LustreError: 166-1: MGC192.168.123.51@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 14007:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LNetError: 120-3: Refusing connection from 192.168.123.51 for 127.0.0.2@tcp: No matching NI LNetError: 13285:0:(socklnd_cb.c:1795:ksocknal_recv_hello()) Error -104 reading HELLO from 127.0.0.2 LNetError: 11b-b: Connection to 127.0.0.2@tcp at host 127.0.0.2:988 was reset: is it running a compatible version of Lustre and is 127.0.0.2@tcp one of its NIDs? Lustre: Skipped 9 previous similar messages Lustre: 13324:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1704197256/real 1704197256] req@ffff8802be086840 x1786980266450112/t0(0) o400->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1704197272 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 | Link to test |
| conf-sanity test 82a: specify OSTs for file (succeed) or directory (succeed) | BUG: unable to handle kernel paging request at ffff8800934b02f8 IP: [<ffffffffa136a23d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f966067 PTE 80000000934b0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs] CPU: 5 PID: 32179 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88025072c9d0 ti: ffff8802b1458000 task.ti: ffff8802b1458000 RIP: 0010:[<ffffffffa136a23d>] [<ffffffffa136a23d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802b145bcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8800a7ba8008 RCX: ffff880275d764eb RDX: 0000000000000005 RSI: ffff8800934b02f0 RDI: ffff8802d3511528 RBP: ffff8802b145bce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8800934b0000 R13: ffff8802b145bcf0 R14: ffffffffa13d02c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800934b02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0387334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0387770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0389395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa039e8be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa039f238>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0382e38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03832c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-114.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 2 previous similar messages Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: new disk, initializing Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 5 previous similar messages Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Skipped 5 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) Lustre: Skipped 14 previous similar messages Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 9 previous similar messages Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-114.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST3e0c-osc-MDT0000: update sequence from 0x13e0c0000 to 0x2c0000402 Lustre: lustre-OSTd1e0-osc-MDT0000: update sequence from 0x1d1e00000 to 0x300000402 Lustre: Mounted lustre-client Lustre: lustre-OSTf116-osc-MDT0000: update sequence from 0x1f1160000 to 0x340000402 Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 5 previous similar messages LustreError: 13616:0:(file.c:246:ll_close_inode_openhandle()) lustre-clilmv-ffff880241b46678: inode [0x200000403:0x1:0x0] mdc close failed: rc = -108 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 13 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 5016:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1703739328 with bad export cookie 5129155615705455403 LustreError: 166-1: MGC192.168.123.116@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 5016:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88028bbf82f8 IP: [<ffffffffa1360078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb9c067 PTE 800000028bbf8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw floppy virtio_blk i2c_core [last unloaded: libcfs] CPU: 3 PID: 12891 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800899a24f0 ti: ffff8802b244c000 task.ti: ffff8802b244c000 RIP: 0010:[<ffffffffa1360078>] [<ffffffffa1360078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802b244fcc8 EFLAGS: 00010246 RAX: ffffffffa13b6820 RBX: ffff8802e3ff4548 RCX: ffff8802839fc5c8 RDX: 0000000000000001 RSI: ffff88028bbf82f0 RDI: ffff8802b244fcf0 RBP: ffff8802b244fce0 R08: ffff8800621b9fe8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000003 R12: ffff88028bbf8000 R13: ffff8802b244fcf0 R14: ffffffffa13c62c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88028bbf82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03ca334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ca770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03cc395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03e18be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03e2238>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03c5e38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c62c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=9398 ... LustreError: 10604:0:(osd_handler.c:694:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 10604:0:(osd_handler.c:694:osd_ro()) Skipped 12 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 146 previous similar messages LustreError: 166-1: MGC192.168.123.78@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 15 previous similar messages LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800733cb440 x1786248476049600/t317827580258(317827580258) o101->lustre-MDT0000-mdc-ffff880242f76678@0@lo:12/10 lens 576/608 e 0 to 0 dl 1703502429 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3945 to 0x380000403:3969 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:3949 to 0x2c0000405:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:3945 to 0x340000404:3969 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:3946 to 0x300000405:3969 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff880242f76678: can't stat MDS #0: rc = -19 Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3 to 0x2c0000403:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4 to 0x380000405:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3 to 0x300000404:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:65 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3 to 0x300000403:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3 to 0x340000403:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:45 to 0x2c0000404:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:4 to 0x380000404:65 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800733cb440 x1786248476049600/t317827580258(317827580258) o101->lustre-MDT0000-mdc-ffff880242f76678@0@lo:12/10 lens 576/608 e 0 to 0 dl 1703502512 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 17 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:4080 to 0x340000404:4129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:4077 to 0x2c0000405:4129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:4080 to 0x300000405:4129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:4087 to 0x380000403:4129 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff880242f76678: can't stat MDS #0: rc = -19 Lustre: server umount lustre-MDT0001 complete LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff880242f76678: can't stat MDS #0: rc = -19 LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff880242f76678: can't stat MDS #0: rc = -19 LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 1 previous similar message LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff880242f76678: can't stat MDS #0: rc = -19 LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 1 previous similar message Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3 to 0x2c0000403:97 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4 to 0x380000405:97 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3 to 0x300000404:97 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3 to 0x340000403:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:4 to 0x380000404:97 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3 to 0x300000403:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:45 to 0x2c0000404:161 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800733cb440 x1786248476049600/t317827580258(317827580258) o101->lustre-MDT0000-mdc-ffff880242f76678@0@lo:12/10 lens 576/608 e 0 to 0 dl 1703502592 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 39 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.78@tcp (at 0@lo) Lustre: Skipped 66 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:4282 to 0x380000403:4321 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:4283 to 0x340000404:4321 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:4282 to 0x2c0000405:4321 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:4282 to 0x300000405:4321 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 11 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 72 previous similar messages LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff880242f76678: can't stat MDS #0: rc = -107 LustreError: 9575:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 2 previous similar messages Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0001: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3 to 0x300000404:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4 to 0x380000405:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3 to 0x2c0000403:129 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3 to 0x300000403:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:45 to 0x2c0000404:193 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:4 to 0x380000404:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3 to 0x340000403:129 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8800733cb440 x1786248476049600/t317827580258(317827580258) o101->lustre-MDT0000-mdc-ffff880242f76678@0@lo:12/10 lens 576/608 e 0 to 0 dl 1703502673 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 7899:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 34 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:4491 to 0x300000405:4513 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:4491 to 0x2c0000405:4513 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:4478 to 0x380000403:4513 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:4502 to 0x340000404:4545 Lustre: DEBUG MARKER: centos-76.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 11 times Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 111e: DNE: unlink striped dir, uncommit on MDT2, fail MDT1/MDT2 | BUG: unable to handle kernel paging request at ffff8800938f82f8 IP: [<ffffffffa1364078>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f964067 PTE 80000000938f8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs] CPU: 13 PID: 26439 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880282a5b760 ti: ffff880275b94000 task.ti: ffff880275b94000 RIP: 0010:[<ffffffffa1364078>] [<ffffffffa1364078>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880275b97cc8 EFLAGS: 00010246 RAX: ffffffffa13ba820 RBX: ffff8800aaf50008 RCX: ffff88029f78bc68 RDX: 0000000000000001 RSI: ffff8800938f82f0 RDI: ffff880275b97cf0 RBP: ffff880275b97ce0 R08: ffff8800858e3528 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000d R12: ffff8800938f8000 R13: ffff880275b97cf0 R14: ffffffffa13ca2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800938f82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03ae334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ae770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b0395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c58be>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c6238>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a9e38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03aa2c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 | Link to test |
| recovery-small test 136: changelog_deregister leaving pending records | BUG: unable to handle kernel paging request at ffff88029f2c82f8 IP: [<ffffffffa136720d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb00067 PTE 800000029f2c8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix serio_raw drm_panel_orientation_quirks floppy libata virtio_blk i2c_core [last unloaded: libcfs] CPU: 8 PID: 7759 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802e6d19280 ti: ffff8802923a4000 task.ti: ffff8802923a4000 RIP: 0010:[<ffffffffa136720d>] [<ffffffffa136720d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802923a7cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88025d014548 RCX: ffff880293c7619f RDX: 0000000000000008 RSI: ffff88029f2c82f0 RDI: ffff88029cdf16a8 RBP: ffff8802923a7ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88029f2c8000 R13: ffff8802923a7cf0 R14: ffffffffa13cb2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88029f2c82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03ef324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ef760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03f1385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa04068ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa0407228>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03eae28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03eb2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDD0000: changelog on Lustre: DEBUG MARKER: recovery-small test_136: @@@@@@ FAIL: Restart of ost4 failed! Lustre: DEBUG MARKER: centos-32.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid,mdc.lustre-MDT0002-mdc-*.mds_server_uuid,osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid,osc.lustre-OST00 Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0002-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: osc.lustre-OST0003-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 6 previous similar messages Lustre: Unmounted lustre-client Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 14717:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1702510992 with bad export cookie 11647614012324697344 LustreError: 14717:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 166-1: MGC192.168.123.34@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 1 previous similar message | Link to test |
| conf-sanity test 105: check file creation for ro and rw bind mnt pt | BUG: unable to handle kernel paging request at ffff8802809d82f8 IP: [<ffffffffa136220d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ebf5067 PTE 80000002809d8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix i2c_core serio_raw virtio_blk libata floppy [last unloaded: libcfs] CPU: 6 PID: 31305 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88009e920010 ti: ffff8802a9dfc000 task.ti: ffff8802a9dfc000 RIP: 0010:[<ffffffffa136220d>] [<ffffffffa136220d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802a9dffcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880240da8008 RCX: ffff880095bcd218 RDX: 0000000000000006 RSI: ffff8802809d82f0 RDI: ffff88031b9e6028 RBP: ffff8802a9dffce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802809d8000 R13: ffff8802a9dffcf0 R14: ffffffffa13c62c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802809d82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a4324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a4760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a6385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03bb8ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03bc228>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039fe28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a02b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 6 previous similar messages Lustre: Unmounted lustre-client Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 14 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 29 previous similar messages Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 29 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 20459:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1702486836 with bad export cookie 1052707251440457041 LustreError: 20459:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: 166-1: MGC192.168.123.16@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0002: haven't heard from client 49a18c13-613a-4c50-bbaf-193fc3ed1a7d (at 0@lo) in 32 seconds. I think it's dead, and I am evicting it. exp ffff8800962d37e8, cur 1702486843 expire 1702486813 last 1702486811 Lustre: server umount lustre-MDT0002 complete Lustre: 31715:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1702486850/real 1702486850] req@ffff8800a05fa0c0 x1785186299030656/t0(0) o39->lustre-MDT0002-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1702486860 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'umount.0' uid:0 gid:0 Lustre: 31715:0:(client.c:2337:ptlrpc_expire_one_request()) Skipped 9 previous similar messages Lustre: server umount lustre-OST0001 complete Lustre: server umount lustre-OST0002 complete Lustre: server umount lustre-OST0003 complete Lustre: DEBUG MARKER: centos-14.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Skipped 3 previous similar messages Lustre: Remounted lustre-client read-only Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 6 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 13 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 5595:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1702486940 with bad export cookie 1052707251440460611 LustreError: 166-1: MGC192.168.123.16@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 11556:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.16@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_78_gf2593fa LNet: Added LNI 192.168.123.16@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-14.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Setting parameter lustre-OST0000.ost.unknown_param in log lustre-OST0000 Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 12759:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1702487002 with bad export cookie 8720349147308819090 LustreError: 166-1: MGC192.168.123.16@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff8800703f82f8 IP: [<ffffffffa1360048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fc02067 PMD 33fa80067 PTE 80000000703f8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw libata i2c_core floppy [last unloaded: libcfs] CPU: 9 PID: 1506 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008595a4f0 ti: ffff880084478000 task.ti: ffff880084478000 RIP: 0010:[<ffffffffa1360048>] [<ffffffffa1360048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88008447bcc8 EFLAGS: 00010246 RAX: ffffffffa13b5820 RBX: ffff8802892467e8 RCX: ffff8803259929b8 RDX: 0000000000000001 RSI: ffff8800703f82f0 RDI: ffff88008447bcf0 RBP: ffff88008447bce0 R08: ffff880242f3d508 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000009 R12: ffff8800703f8000 R13: ffff88008447bcf0 R14: ffffffffa13c42c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800703f82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03be324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03be760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c0385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d58ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d6228>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b9e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03ba2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=7616 ... LustreError: 8672:0:(osd_handler.c:694:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** LustreError: 8672:0:(osd_handler.c:694:osd_ro()) Skipped 13 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 166-1: MGC192.168.123.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 15 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 8 previous similar messages LustreError: 9797:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88006fb5f540 x1785138135889536/t317827580263(317827580263) o101->lustre-MDT0000-mdc-ffff88028c50efc8@0@lo:12/10 lens 624/608 e 0 to 0 dl 1702443555 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 9797:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x380000403:3946 to 0x380000403:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x340000403:3945 to 0x340000403:3969 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:4014 to 0x2c0000403:4033 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:3946 to 0x300000405:3969 Lustre: DEBUG MARKER: centos-101.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff8802be1302f8 IP: [<ffffffffa136f048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea09067 PTE 80000002be130060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy i2c_core libata [last unloaded: libcfs] CPU: 12 PID: 32038 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8803208d24f0 ti: ffff880288454000 task.ti: ffff880288454000 RIP: 0010:[<ffffffffa136f048>] [<ffffffffa136f048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880288457cc8 EFLAGS: 00010246 RAX: ffffffffa13c4820 RBX: ffff8802894833f8 RCX: ffff8802936d5598 RDX: 0000000000000001 RSI: ffff8802be1302f0 RDI: ffff880288457cf0 RBP: ffff880288457ce0 R08: ffff8800927d8558 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000c R12: ffff8802be130000 R13: ffff880288457cf0 R14: ffffffffa13d32c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802be1302f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03d6324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03d6760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03d8385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ed8ae>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ee228>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03d1e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03d22b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=990 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-mdc-ffff8802c76c4138: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 15 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802c76c4138: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 57 previous similar messages LustreError: 1179:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0000-mdc-ffff8802c76c4138: can't stat MDS #0: rc = -107 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 227 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.83@tcp (at 0@lo) Lustre: Skipped 56 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 5 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3930 to 0x380000403:3969 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3930 to 0x300000403:3969 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:3969 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 LustreError: 18850:0:(ldlm_lockd.c:1495:ldlm_handle_enqueue()) ### lock on destroyed export ffff8800acb737e8 ns: mdt-lustre-MDT0001_UUID lock: ffff8802519e6580/0x49bee6e6be486d1f lrc: 3/0,0 mode: CW/CW res: [0x2400007ed:0xe5:0x0].0x0 bits 0x5/0x0 rrc: 2 type: IBT gid 0 flags: 0x50200000000000 nid: 0@lo remote: 0x49bee6e6be486d11 expref: 4 pid: 18850 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete LustreError: 31842:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880089d313c0 x1785090145807488/t4294967786(4294967786) o101->lustre-MDT0001-mdc-ffff8802c76c4138@0@lo:12/10 lens 624/608 e 0 to 0 dl 1702398029 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 31842:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:57 to 0x300000404:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:58 to 0x340000404:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:51 to 0x2c0000404:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:61 to 0x380000405:129 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 | Link to test |
| replay-single test 70c: tar 3mdts recovery | BUG: unable to handle kernel paging request at ffff88028aef82f8 IP: [<ffffffffa135b048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eba2067 PTE 800000028aef8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy i2c_core serio_raw virtio_blk libata [last unloaded: libcfs] CPU: 0 PID: 31972 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88027895c9d0 ti: ffff8800a0ec8000 task.ti: ffff8800a0ec8000 RIP: 0010:[<ffffffffa135b048>] [<ffffffffa135b048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8800a0ecbcc8 EFLAGS: 00010246 RAX: ffffffffa13b0820 RBX: ffff88026e3f67e8 RCX: ffff88028d04c458 RDX: 0000000000000001 RSI: ffff88028aef82f0 RDI: ffff8800a0ecbcf0 RBP: ffff8800a0ecbce0 R08: 0000000000000000 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000000 R12: ffff88028aef8000 R13: ffff8800a0ecbcf0 R14: ffffffffa13bf2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88028aef82f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa03b3324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b3760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b5385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ca8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cb278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03aee28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03af2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70c fail mds3 1 times Lustre: Failing over lustre-MDT0002 | Link to test |
| replay-single test 102c: check replay w/o reconstruction with multiple mod RPCs in flight | BUG: unable to handle kernel paging request at ffff8801ef2b02f8 IP: [<ffffffffa135c048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33f1fd067 PMD 33f083067 PTE 80000001ef2b0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks floppy virtio_blk serio_raw ata_piix i2c_core libata [last unloaded: libcfs] CPU: 2 PID: 20584 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88024871b760 ti: ffff88008d428000 task.ti: ffff88008d428000 RIP: 0010:[<ffffffffa135c048>] [<ffffffffa135c048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88008d42bcc8 EFLAGS: 00010246 RAX: ffffffffa13b1820 RBX: ffff880324b05698 RCX: ffff8802a8e0faf8 RDX: 0000000000000001 RSI: ffff8801ef2b02f0 RDI: ffff88008d42bcf0 RBP: ffff88008d42bce0 R08: ffff880260348008 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000002 R12: ffff8801ef2b0000 R13: ffff88008d42bcf0 R14: ffffffffa13c02c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8801ef2b02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03ba324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ba760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03bc385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d18fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d2278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b5e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b62b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 115: failover for create/unlink striped directory | BUG: unable to handle kernel paging request at ffff8802907782f8 IP: [<ffffffffa1368048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eb76067 PTE 8000000290778060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console i2c_piix4 pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks floppy libata i2c_core serio_raw virtio_blk [last unloaded: libcfs] CPU: 14 PID: 18726 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880288d1b760 ti: ffff88028b9d8000 task.ti: ffff88028b9d8000 RIP: 0010:[<ffffffffa1368048>] [<ffffffffa1368048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88028b9dbcc8 EFLAGS: 00010246 RAX: ffffffffa13bd820 RBX: ffff880094dfe7e8 RCX: ffff88027feabdd8 RDX: 0000000000000001 RSI: ffff8802907782f0 RDI: ffff88028b9dbcf0 RBP: ffff88028b9dbce0 R08: ffff8802c7833fc8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000e R12: ffff880290778000 R13: ffff88028b9dbcf0 R14: ffffffffa13cc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802907782f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa039f324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa039f760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a1385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b68fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b7278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039ae28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039b2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 81h: DNE: unlink remote dir, drop request reply, fail 2 MDTs | BUG: unable to handle kernel paging request at ffff8802f2fc82f8 IP: [<ffffffffa13b1048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e861067 PTE 80000002f2fc8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix floppy serio_raw drm_panel_orientation_quirks virtio_blk libata i2c_core [last unloaded: libcfs] CPU: 4 PID: 842 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800ace3b760 ti: ffff88026ed88000 task.ti: ffff88026ed88000 RIP: 0010:[<ffffffffa13b1048>] [<ffffffffa13b1048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88026ed8bcc8 EFLAGS: 00010246 RAX: ffffffffa1406820 RBX: ffff88027af41158 RCX: ffff880231099428 RDX: 0000000000000001 RSI: ffff8802f2fc82f0 RDI: ffff88026ed8bcf0 RBP: ffff88026ed8bce0 R08: ffff8803273f2538 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000001 R12: ffff8802f2fc8000 R13: ffff88026ed8bcf0 R14: ffffffffa14152c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802f2fc82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c6324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c6760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c8385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03dd8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03de278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03c1e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c22b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: *** cfs_fail_loc=119, val=2147483648*** Lustre: Skipped 6 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 589:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701885207 with bad export cookie 7669731948177914445 Lustre: Failing over lustre-MDT0001 LustreError: 589:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 8 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | BUG: unable to handle kernel paging request at ffff880284c082f8 IP: [<ffffffffa136120d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ebd3067 PTE 8000000284c08060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata serio_raw i2c_core virtio_blk floppy [last unloaded: libcfs] CPU: 5 PID: 19115 Comm: kworker/u32:5 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800778fb760 ti: ffff8802a92dc000 task.ti: ffff8802a92dc000 RIP: 0010:[<ffffffffa136120d>] [<ffffffffa136120d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802a92dfcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880246e033f8 RCX: ffff8803221b019f RDX: 0000000000000005 RSI: ffff880284c082f0 RDI: ffff88009dd8f248 RBP: ffff8802a92dfce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880284c08000 R13: ffff8802a92dfcf0 R14: ffffffffa13c52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880284c082f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a0324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a0760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a2385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b78fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b8278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039be28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039c2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:1640 to 0x340000404:1665 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:1672 to 0x300000403:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:1672 to 0x380000404:1697 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:1673 to 0x2c0000405:1697 Lustre: DEBUG MARKER: centos-56.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 38: MDS recreates missing lov_objid file from OST data | BUG: unable to handle kernel paging request at ffff880285b102f8 IP: [<ffffffffa134820d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ebcc067 PTE 8000000285b10060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ttm ata_generic drm pata_acpi drm_panel_orientation_quirks ata_piix floppy serio_raw virtio_blk i2c_core libata [last unloaded: libcfs] CPU: 0 PID: 17446 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802b87449d0 ti: ffff8802c7f94000 task.ti: ffff8802c7f94000 RIP: 0010:[<ffffffffa134820d>] [<ffffffffa134820d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802c7f97cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880260b9a2a8 RCX: ffff8802b4bea03a RDX: 0000000000000000 RSI: ffff880285b102f0 RDI: ffff88009be5ca88 RBP: ffff8802c7f97ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880285b10000 R13: ffff8802c7f97cf0 R14: ffffffffa13ac2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880285b102f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa03be324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03be760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c0385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d58fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d6278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b9e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03ba2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 28749:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701744026 with bad export cookie 13208397328852409408 LustreError: 166-1: MGC192.168.123.21@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 30111:0:(osp_object.c:637:osp_attr_get()) lustre-MDT0001-osp-MDT0002: osp_attr_get update error [0x240000bd2:0x1:0x0]: rc = -5 LustreError: 30111:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802ac9e86c0 x1784407786598144/t0(0) o1000->lustre-MDT0001-osp-MDT0002@0@lo:24/4 lens 264/4320 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 30111:0:(llog_cat.c:737:llog_cat_cancel_arr_rec()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 llog-records: rc = -5 LustreError: 30111:0:(llog_cat.c:773:llog_cat_cancel_records()) lustre-MDT0001-osp-MDT0002: fail to cancel 1 of 1 llog-records: rc = -5 Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: delete lov_objid file on MDS Lustre: 16624:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701744031/real 1701744031] req@ffff8802ac9ece40 x1784407786598400/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1701744047 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0002-lwp-OST0000: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: 16625:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701744036/real 1701744036] req@ffff8802ac9ebac0 x1784407786598464/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1701744052 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000bd0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:42 to 0x2c0000401:65 Lustre: 16615:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701744041/real 1701744041] req@ffff8802ac9e8d40 x1784407786598592/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1701744057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:65 Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: 16615:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701744046/real 1701744046] req@ffff8802ac9ee840 x1784407786598656/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1701744062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0002-lwp-OST0000: Connection restored to 192.168.123.21@tcp (at 0@lo) Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.123.21@tcp (at 0@lo) Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 30988:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701744071 with bad export cookie 13208397328852412320 LustreError: 30988:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages LustreError: 166-1: MGC192.168.123.21@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 13 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete LustreError: 5127:0:(import.c:355:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) LustreError: 5127:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff880272f086c0 x1784407786628288/t0(0) o250->MGC192.168.123.21@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1701744081 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 5127:0:(import.c:389:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. LustreError: 16609:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802c978e1c0 x1784407786629568/t0(0) o250->MGC192.168.123.21@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000bd0:3 to 0x2c0000bd0:33 Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:42 to 0x2c0000401:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:35 to 0x2c0000400:97 Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.123.21@tcp (at 0@lo) Lustre: 16621:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701744076/real 1701744076] req@ffff8802c978d4c0 x1784407786629376/t0(0) o400->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1701744092 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-19.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 28742:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701744104 with bad export cookie 13208397328852414861 LustreError: 28742:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 3 previous similar messages LustreError: 166-1: MGC192.168.123.21@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 9 previous similar messages | Link to test |
| sanity-pfl test complete, duration 2308 sec | BUG: unable to handle kernel paging request at ffff88015a9b02f8 IP: [<ffffffffa13ef048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33f5ff067 PMD 33f52a067 PTE 800000015a9b0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) nfsd ext4 mbcache jbd2 loop zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix virtio_blk libata serio_raw i2c_core floppy [last unloaded: libcfs] CPU: 11 PID: 2922 Comm: kworker/u32:0 Kdump: loaded Tainted: P W OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88012aed24f0 ti: ffff8802ca3fc000 task.ti: ffff8802ca3fc000 RIP: 0010:[<ffffffffa13ef048>] [<ffffffffa13ef048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802ca3ffcc8 EFLAGS: 00010246 RAX: ffffffffa1444820 RBX: ffff8800a13933f8 RCX: ffff88016b8a1fa8 RDX: 0000000000000001 RSI: ffff88015a9b02f0 RDI: ffff8802ca3ffcf0 RBP: ffff8802ca3ffce0 R08: ffff880013c08558 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000b R12: ffff88015a9b0000 R13: ffff8802ca3ffcf0 R14: ffffffffa14532c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88015a9b02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03d2324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03d2760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03d4385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03e98fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03ea278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03cde28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03ce2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 17 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 62 previous similar messages LustreError: 12658:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701532672 with bad export cookie 16236896660468856444 LustreError: 12646:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701532674 with bad export cookie 16236896660468856437 LustreError: 12646:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 12 previous similar messages LustreError: 166-1: MGC192.168.123.40@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.40@tcp (at 0@lo) Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-OST0002: deleting orphan objects from 0x34000040d:1237 to 0x34000040d:1281 Lustre: lustre-OST0001: deleting orphan objects from 0x30000040d:1237 to 0x30000040d:1281 Lustre: lustre-OST0000: deleting orphan objects from 0x2c000040d:1205 to 0x2c000040d:1249 Lustre: lustre-OST0003: deleting orphan objects from 0x38000040d:1237 to 0x38000040d:1281 Lustre: DEBUG MARKER: centos-38.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 32548:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701532707 with bad export cookie 16236896660469012292 LustreError: 32548:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 6 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages | Link to test |
| replay-single test 100c: DNE: create striped dir, abort_recov_mdt mds2 | BUG: unable to handle kernel paging request at ffff8802da9982f8 IP: [<ffffffffa133b048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e924067 PTE 80000002da998060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console i2c_piix4 virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw floppy i2c_core libata [last unloaded: libcfs] CPU: 1 PID: 19238 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802e1f35c40 ti: ffff8803200a4000 task.ti: ffff8803200a4000 RIP: 0010:[<ffffffffa133b048>] [<ffffffffa133b048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8803200a7cc8 EFLAGS: 00010246 RAX: ffffffffa1390820 RBX: ffff8802d00833f8 RCX: ffff8802bcec5428 RDX: 0000000000000002 RSI: ffff8802da9982f0 RDI: ffff8803200a7cf0 RBP: ffff8803200a7ce0 R08: ffff8800a58a6a48 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000001 R12: ffff8802da998000 R13: ffff8803200a7cf0 R14: ffffffffa139f2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802da9982f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b3324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b3760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b5385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ca8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cb278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03aee28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03af2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 342:0:(osd_handler.c:694:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** LustreError: 342:0:(osd_handler.c:694:osd_ro()) Skipped 21 previous similar messages Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 66: replace nids | BUG: unable to handle kernel paging request at ffff8802ae5902f8 IP: [<ffffffffa133c20d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea87067 PTE 80000002ae590060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic drm_kms_helper pata_acpi ttm drm drm_panel_orientation_quirks floppy ata_piix serio_raw i2c_core virtio_blk libata [last unloaded: libcfs] CPU: 3 PID: 8581 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880329025c40 ti: ffff88024d8ac000 task.ti: ffff88024d8ac000 RIP: 0010:[<ffffffffa133c20d>] [<ffffffffa133c20d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88024d8afcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88005bb60008 RCX: ffff8802d919ca9e RDX: 0000000000000003 RSI: ffff8802ae5902f0 RDI: ffff8802c6540188 RBP: ffff88024d8afce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802ae590000 R13: ffff88024d8afcf0 R14: ffffffffa13a02c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331ac0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802ae5902f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03ec324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03ec760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03ee385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa04038fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa0404278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03e7e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03e82b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Permanently deactivating lustre-OST0000 Lustre: Setting parameter lustre-OST0000-osc.osc.active in log lustre-client Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request LustreError: 18577:0:(mgs_llog.c:1612:mgs_replace_nids()) Only MGS is allowed to be started LustreError: 18577:0:(mgs_handler.c:1099:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 Lustre: Unmounted lustre-client LustreError: 18603:0:(mgs_llog.c:1612:mgs_replace_nids()) Only MGS is allowed to be started LustreError: 18603:0:(mgs_handler.c:1099:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 Lustre: server umount lustre-OST0000 complete LustreError: 18780:0:(mgs_llog.c:1612:mgs_replace_nids()) Only MGS is allowed to be started LustreError: 18780:0:(mgs_handler.c:1099:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: 18802:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701401576/real 1701401576] req@ffff8802f35fc140 x1784048711163584/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1701401586 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 15434:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1701401588 with bad export cookie 17792734165227007247 LustreError: 15434:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: 18987:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701401588/real 1701401588] req@ffff88020209a740 x1784048711165632/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1701401598 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete Lustre: 19216:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1701401598/real 1701401598] req@ffff8802f35fc140 x1784048711166720/t0(0) o9->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1701401608 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 | Link to test |
| replay-single test 115: failover for create/unlink striped directory | BUG: unable to handle kernel paging request at ffff8800892c02f8 IP: [<ffffffffa1351048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f9b7067 PTE 80000000892c0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon i2c_piix4 virtio_console pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix serio_raw virtio_blk floppy i2c_core libata [last unloaded: libcfs] CPU: 7 PID: 11320 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88022bf3dc40 ti: ffff88029ef70000 task.ti: ffff88029ef70000 RIP: 0010:[<ffffffffa1351048>] [<ffffffffa1351048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88029ef73cc8 EFLAGS: 00010246 RAX: ffffffffa13a6820 RBX: ffff8802ae3ac548 RCX: ffff8802e666db58 RDX: 0000000000000001 RSI: ffff8800892c02f0 RDI: ffff88029ef73cf0 RBP: ffff88029ef73ce0 R08: ffff8802bc9864f8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000007 R12: ffff8800892c0000 R13: ffff88029ef73cf0 R14: ffffffffa13b52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800892c02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03cc324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03cc760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03ce385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03e38fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03e4278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03c7e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c82b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:1310 to 0x300000405:1729 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:1310 to 0x380000405:1729 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:1341 to 0x340000404:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:1310 to 0x2c0000405:1729 Lustre: DEBUG MARKER: centos-41.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: Failing over lustre-MDT0002 | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88031ece82f8 IP: [<ffffffffa1373048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 23c4067 PMD 33feeb067 PTE 800000031ece8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs] CPU: 0 PID: 30412 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88028c0649d0 ti: ffff880252acc000 task.ti: ffff880252acc000 RIP: 0010:[<ffffffffa1373048>] [<ffffffffa1373048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880252acfcc8 EFLAGS: 00010246 RAX: ffffffffa13c8820 RBX: ffff88029ae3b3f8 RCX: ffff880321615878 RDX: 0000000000000001 RSI: ffff88031ece82f0 RDI: ffff880252acfcf0 RBP: ffff880252acfce0 R08: ffff88027dfeafd8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000000 R12: ffff88031ece8000 R13: ffff880252acfcf0 R14: ffffffffa13d72c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88031ece82f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa03aa324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03aa760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03ac385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03c18fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03c2278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03a5e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03a62b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=29614 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 17 previous similar messages Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 57 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 202 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 55 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:3934 to 0x2c0000404:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3930 to 0x340000405:3969 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:3930 to 0x300000405:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3930 to 0x380000403:3969 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 30069:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88008e7b8040 x1783724997532928/t4294967787(4294967787) o101->lustre-MDT0001-mdc-ffff88029e2853d8@0@lo:12/10 lens 624/608 e 0 to 0 dl 1701096177 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 30069:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:60 to 0x2c0000403:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:59 to 0x340000404:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:60 to 0x380000405:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:59 to 0x300000403:129 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 | Link to test |
| replay-single test 80h: DNE: create remote dir, drop MDT1 rep, fail 2 MDTs | BUG: unable to handle kernel paging request at ffff8803231502f8 IP: [<ffffffffa135d048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 23c4067 PMD 33fec9067 PTE 8000000323150060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw libata i2c_core virtio_blk [last unloaded: libcfs] CPU: 14 PID: 8793 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88008970a4f0 ti: ffff8802880a8000 task.ti: ffff8802880a8000 RIP: 0010:[<ffffffffa135d048>] [<ffffffffa135d048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802880abcc8 EFLAGS: 00010246 RAX: ffffffffa13b2820 RBX: ffff88023c6ca2a8 RCX: ffff880280f6f0e8 RDX: 0000000000000001 RSI: ffff8803231502f0 RDI: ffff8802880abcf0 RBP: ffff8802880abce0 R08: ffff880326623528 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000e R12: ffff880323150000 R13: ffff8802880abcf0 R14: ffffffffa13c12c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8803231502f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c1324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c1760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c3385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d88fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d9278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03bce28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03bd2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: *** cfs_fail_loc=119, val=2147483648*** Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 LustreError: 28318:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700985514 with bad export cookie 17058363440524172838 LustreError: 28318:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 1 previous similar message | Link to test |
| conf-sanity test 57a: initial registration from failnode should fail (should return errs) | BUG: unable to handle kernel paging request at ffff8802756c82f8 IP: [<ffffffffa137820d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ec4f067 PTE 80000002756c8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm drm_panel_orientation_quirks ata_piix floppy i2c_core virtio_blk serio_raw libata [last unloaded: libcfs] CPU: 11 PID: 12214 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880247b2b760 ti: ffff8802f955c000 task.ti: ffff8802f955c000 RIP: 0010:[<ffffffffa137820d>] [<ffffffffa137820d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802f955fcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8800aeb7b3f8 RCX: ffff8803232f219f RDX: 000000000000000b RSI: ffff8802756c82f0 RDI: ffff88022c060588 RBP: ffff8802f955fce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802756c8000 R13: ffff8802f955fcf0 R14: ffffffffa13dc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802756c82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b5324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b5760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b7385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03cc8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cd278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b0e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b12b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-34.localnet: executing load_modules_local Lustre: DEBUG MARKER: centos-34.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) Lustre: Skipped 10 previous similar messages Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 6 sec Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-34.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 26988:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700910430 with bad export cookie 11965261058678359015 LustreError: 166-1: MGC192.168.123.36@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 26988:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 5 previous similar messages | Link to test |
| replay-dual test 26: dbench and tar with mds failover | BUG: unable to handle kernel paging request at ffff8802f4b882f8 IP: [<ffffffffa135620d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e853067 PTE 80000002f4b88060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm drm_panel_orientation_quirks i2c_core virtio_blk floppy serio_raw libata [last unloaded: libcfs] CPU: 15 PID: 8485 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031ec30010 ti: ffff880291e18000 task.ti: ffff880291e18000 RIP: 0010:[<ffffffffa135620d>] [<ffffffffa135620d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff880291e1bcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880256d99158 RCX: ffff8800845c8dd6 RDX: 000000000000000f RSI: ffff8802f4b882f0 RDI: ffff880322e71a28 RBP: ffff880291e1bce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802f4b88000 R13: ffff880291e1bcf0 R14: ffffffffa13ba2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802f4b882f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c4324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c4760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c6385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03db8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03dc278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03bfe28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c02b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 6558:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700895737 with bad export cookie 13010833291809349630 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 786 previous similar messages Lustre: Skipped 200 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection restored to 192.168.123.3@tcp (at 0@lo) Lustre: Skipped 198 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1670 to 0x340000403:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1671 to 0x300000404:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1670 to 0x380000403:1697 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1672 to 0x2c0000403:1697 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:236 to 0x2c0000404:257 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:236 to 0x380000404:257 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:237 to 0x300000403:257 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:236 to 0x340000405:257 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete LustreError: 703:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802f1fe61c0 x1783516351290688/t4294968839(4294968839) o101->lustre-MDT0002-mdc-ffff880086316678@0@lo:12/10 lens 592/608 e 0 to 0 dl 1700895844 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'tar.0' uid:0 gid:0 LustreError: 703:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 53 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:32 to 0x300000405:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:32 to 0x380000405:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:32 to 0x2c0000405:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:32 to 0x340000404:65 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1729 to 0x300000404:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1727 to 0x2c0000403:1761 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1729 to 0x340000403:1761 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1729 to 0x380000403:1761 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: Failing over lustre-MDT0001 LustreError: 11-0: lustre-MDT0001-mdc-ffff88031c4d4a88: operation mds_readpage to node 0@lo failed: rc = -107 LustreError: Skipped 42 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:278 to 0x2c0000404:321 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:278 to 0x340000405:321 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:278 to 0x380000404:321 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:278 to 0x300000403:321 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002: Recovery over after 0:06, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 23 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:360 to 0x380000405:385 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:361 to 0x2c0000405:385 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:360 to 0x340000404:385 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:361 to 0x300000405:385 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: Skipped 22 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1787 to 0x380000403:1825 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1795 to 0x2c0000403:1825 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1799 to 0x340000403:1825 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1794 to 0x300000404:1825 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff88031c4d4a88: can't stat MDS #0: rc = -19 Lustre: server umount lustre-MDT0001 complete LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff88031c4d4a88: can't stat MDS #0: rc = -19 LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff88031c4d4a88: can't stat MDS #0: rc = -19 LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff88031c4d4a88: can't stat MDS #0: rc = -19 LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 1 previous similar message LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff88031c4d4a88: can't stat MDS #0: rc = -19 LustreError: 24673:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 1 previous similar message Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:329 to 0x380000404:353 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:328 to 0x300000403:353 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:326 to 0x2c0000404:353 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:324 to 0x340000405:353 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:458 to 0x380000405:481 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:472 to 0x2c0000405:513 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:483 to 0x340000404:513 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:466 to 0x300000405:481 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 6094:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880089f34e40 x1783516355585344/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 LustreError: 6094:0:(client.c:1281:ptlrpc_import_delay_req()) Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1899 to 0x300000404:1921 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1898 to 0x380000403:1921 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1898 to 0x340000403:1921 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1898 to 0x2c0000403:1921 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 21370:0:(osd_handler.c:694:osd_ro()) lustre-MDT0001: *** setting device osd-zfs read-only *** LustreError: 21370:0:(osd_handler.c:694:osd_ro()) Skipped 14 previous similar messages Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 11 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 6557:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700896043 with bad export cookie 13010833291809929587 LustreError: 14245:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0001-mdc-ffff88031c4d4a88: can't stat MDS #0: rc = -107 LustreError: 14245:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 2 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:383 to 0x2c0000404:417 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:382 to 0x340000405:417 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:383 to 0x380000404:417 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:382 to 0x300000403:417 Lustre: 6692:0:(mdt_recovery.c:149:mdt_req_from_lrd()) @@@ restoring transno req@ffff8802e0593440 x1783516356052352/t42949673692(0) o36->e8ebdc07-3cca-4ba2-b8a1-b94dafb23e18@0@lo:384/0 lens 560/2880 e 0 to 0 dl 1700896094 ref 1 fl Interpret:/202/0 rc 0/0 job:'tar.0' uid:0 gid:0 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 12 times Lustre: Failing over lustre-MDT0002 LustreError: 6237:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880325eb61c0 x1783516356412032/t0(0) o105->lustre-MDT0002@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002: in recovery but waiting for the first client to connect Lustre: Skipped 21 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:515 to 0x300000405:545 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:550 to 0x2c0000405:577 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:544 to 0x340000404:577 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:514 to 0x380000405:545 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 13 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 166-1: MGC192.168.123.3@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 10 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:1945 to 0x2c0000403:1985 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1946 to 0x300000404:1985 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:1945 to 0x380000403:1985 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:1946 to 0x340000403:1985 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 14 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:450 to 0x300000403:481 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:451 to 0x340000405:481 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:451 to 0x2c0000404:481 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:451 to 0x380000404:481 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 15 times Lustre: Failing over lustre-MDT0002 Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 48 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:601 to 0x2c0000405:641 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:600 to 0x340000404:641 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:567 to 0x380000405:609 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:566 to 0x300000405:609 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 16 times Lustre: Failing over lustre-MDT0000 LustreError: 6573:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff880073ef0d40 x1783516357889728/t0(0) o105->lustre-MDT0000@0@lo:15/16 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:2006 to 0x300000404:2049 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:2007 to 0x380000403:2049 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:2005 to 0x340000403:2049 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:2004 to 0x2c0000403:2049 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 17 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:504 to 0x380000404:545 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:507 to 0x340000405:545 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:520 to 0x300000403:545 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:501 to 0x2c0000404:545 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 18 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 18 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:664 to 0x2c0000405:705 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:637 to 0x380000405:673 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:643 to 0x300000405:673 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:679 to 0x340000404:705 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 19 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:2090 to 0x300000404:2113 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:2078 to 0x340000403:2113 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:2078 to 0x380000403:2113 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:2094 to 0x2c0000403:2113 Lustre: DEBUG MARKER: centos-1.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 20 times Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 57a: initial registration from failnode should fail (should return errs) | BUG: unable to handle kernel paging request at ffff88023e1a02f8 IP: [<ffffffffa138920d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33effc067 PMD 33ee0b067 PTE 800000023e1a0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console virtio_balloon i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy libata serio_raw virtio_blk i2c_core [last unloaded: libcfs] CPU: 13 PID: 10586 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88009a3cdc40 ti: ffff88026fc5c000 task.ti: ffff88026fc5c000 RIP: 0010:[<ffffffffa138920d>] [<ffffffffa138920d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88026fc5fcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88025097b3f8 RCX: ffff88031e3fe218 RDX: 000000000000000d RSI: ffff88023e1a02f0 RDI: ffff880271c5be68 RBP: ffff88026fc5fce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88023e1a0000 R13: ffff88026fc5fcf0 R14: ffffffffa13ed2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88023e1a02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03bf324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03bf760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c1385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d68fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d7278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03bae28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03bb2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-54.localnet: executing load_modules_local Lustre: DEBUG MARKER: centos-54.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: Skipped 6 previous similar messages Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-54.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 10 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages LustreError: 1427:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700850335 with bad export cookie 14735533311217891842 LustreError: 166-1: MGC192.168.123.56@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88028aee82f8 IP: [<ffffffffa1350048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33eba2067 PTE 800000028aee8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm ata_piix drm floppy drm_panel_orientation_quirks serio_raw libata virtio_blk i2c_core [last unloaded: libcfs] CPU: 5 PID: 9190 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c0f91280 ti: ffff8803232e4000 task.ti: ffff8803232e4000 RIP: 0010:[<ffffffffa1350048>] [<ffffffffa1350048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8803232e7cc8 EFLAGS: 00010246 RAX: ffffffffa13a5820 RBX: ffff8800ac86e7e8 RCX: ffff88007b9b5fa8 RDX: 0000000000000001 RSI: ffff88028aee82f0 RDI: ffff8803232e7cf0 RBP: ffff8803232e7ce0 R08: ffff8802f3985508 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000005 R12: ffff88028aee8000 R13: ffff8803232e7cf0 R14: ffffffffa13b42c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88028aee82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c4324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c4760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c6385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03db8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03dc278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03bfe28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03c02b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=12785 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 25 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 58 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 193 previous similar messages LustreError: 166-1: MGC192.168.123.28@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 11 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 7 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.28@tcp (at 0@lo) Lustre: Skipped 56 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3930 to 0x300000403:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3930 to 0x380000403:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:3969 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:3969 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:3 to 0x2c0000405:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:3 to 0x300000405:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4 to 0x380000405:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3 to 0x340000405:65 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0002 complete LustreError: 14435:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8801f48d2dc0 x1783402389362688/t4294967854(4294967854) o101->lustre-MDT0002-mdc-ffff8802f5cb8008@0@lo:12/10 lens 576/608 e 0 to 0 dl 1700788538 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 14435:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:162 to 0x2c0000404:193 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:120 to 0x300000404:193 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:110 to 0x380000404:193 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:115 to 0x340000404:193 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3930 to 0x380000403:4001 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:3930 to 0x300000403:4001 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:4001 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:4001 Lustre: DEBUG MARKER: centos-26.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 100d: DNE: cancel update logs upon recovery abort | BUG: unable to handle kernel paging request at ffff8802436e82f8 IP: [<ffffffffa135a048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33eddf067 PTE 80000002436e8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks libata i2c_core virtio_blk serio_raw floppy [last unloaded: libcfs] CPU: 12 PID: 32323 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880085ad0010 ti: ffff88028f280000 task.ti: ffff88028f280000 RIP: 0010:[<ffffffffa135a048>] [<ffffffffa135a048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff88028f283cc8 EFLAGS: 00010246 RAX: ffffffffa13af820 RBX: ffff880231051158 RCX: ffff880240fa4b88 RDX: 0000000000000001 RSI: ffff8802436e82f0 RDI: ffff88028f283cf0 RBP: ffff88028f283ce0 R08: ffff88027e876a48 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000c R12: ffff8802436e8000 R13: ffff88028f283cf0 R14: ffffffffa13be2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802436e82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a0324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a0760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a2385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b78fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b8278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039be28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039c2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: Failing over lustre-MDT0001 | Link to test |
| conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one | BUG: unable to handle kernel paging request at ffff8800832282f8 IP: [<ffffffffa137120d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f9e7067 PTE 8000000083228060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_console virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw virtio_blk libata i2c_core [last unloaded: libcfs] CPU: 11 PID: 28547 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802b3ca0010 ti: ffff8802d3770000 task.ti: ffff8802d3770000 RIP: 0010:[<ffffffffa137120d>] [<ffffffffa137120d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802d3773cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8800a7468008 RCX: ffff8802c4c153fb RDX: 000000000000000b RSI: ffff8800832282f0 RDI: ffff8802c17b6bf8 RBP: ffff8802d3773ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880083228000 R13: ffff8802d3773cf0 R14: ffffffffa13d52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331cc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800832282f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03d9324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03d9760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03db385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03f08fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03f1278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03d4e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03d52b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Key type lgssc unregistered LNet: 14383:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:33 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 15373:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748141 with bad export cookie 8312100449826595729 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 19374:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 20486:0:(libcfs_fail.h:190:cfs_race()) cfs_race id 716 sleeping LustreError: 20488:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 716 waking LustreError: 20486:0:(libcfs_fail.h:199:cfs_race()) cfs_fail_race id 716 awake: rc=4935 LustreError: 20486:0:(tgt_mount.c:2183:server_fill_super()) Unable to start osd on lustre-mdt1/mdt1: -114 LustreError: 20486:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -114 LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing lsmod LustreError: 22338:0:(libcfs_fail.h:190:cfs_race()) cfs_race id 716 sleeping LustreError: 22337:0:(libcfs_fail.h:201:cfs_race()) cfs_fail_race id 716 waking LustreError: 22338:0:(libcfs_fail.h:199:cfs_race()) cfs_fail_race id 716 awake: rc=4988 LustreError: 22338:0:(tgt_mount.c:2183:server_fill_super()) Unable to start osd on lustre-ost1/ost1: -114 LustreError: 22338:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -114 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 20510:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748197 with bad export cookie 16336726935417322144 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 21163:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff88029c8ddb40 x1783363704731200/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 21747:0:(client.c:1281:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@ffff8802f8be93c0 x1783363704732608/t0(0) o900->lustre-MDT0000-lwp-MDT0002@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0002 complete Lustre: 23213:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1700748208/real 1700748208] req@ffff8802f8bef540 x1783363704733056/t0(0) o39->lustre-MDT0002-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1700748218 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-OST0000 complete LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:3 to 0x2c0000400:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 23728:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748267 with bad export cookie 16336726935417323180 LustreError: 23728:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 27542:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:67 to 0x2c0000400:97 Lustre: Mounted lustre-client Lustre: Setting parameter lustre-client.llite.some_wrong_param in log lustre-client Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: Modifying parameter lustre-client.llite.some_wrong_param in log lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 28653:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748353 with bad export cookie 18200754815407670042 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 32551:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:99 to 0x2c0000400:129 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 1163:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748437 with bad export cookie 14967538085560936545 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 5127:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp lustre: Unknown symbol range_unlock (err 0) lustre: Unknown symbol lprocfs_stats_clear (err 0) lustre: Unknown symbol lprocfs_wr_nosquash_nids (err 0) lustre: Unknown symbol cl_io_end (err 0) lustre: Unknown symbol cl_page_list_fini (err 0) lustre: Unknown symbol req_capsule_field_present (err 0) lustre: Unknown symbol cl_object_attr_get (err 0) lustre: Unknown symbol lmv_stripe_object_put (err 0) lustre: Unknown symbol cl_page_assume (err 0) lustre: Unknown symbol cl_page_get (err 0) lustre: Unknown symbol class_manual_cleanup (err 0) lustre: Unknown symbol range_lock_init (err 0) lustre: Unknown symbol cl_object_maxbytes (err 0) lustre: Unknown symbol lu_object_header_print (err 0) lustre: Unknown symbol RMF_EADATA (err 0) lustre: Unknown symbol RMF_MDT_MD (err 0) lustre: Unknown symbol lustre_swab_lmv_user_md (err 0) lustre: Unknown symbol cl_site_init (err 0) lustre: Unknown symbol _ldlm_lock_debug (err 0) lustre: Unknown symbol cl_cache_init (err 0) lustre: Unknown symbol lu_object_add (err 0) lustre: Unknown symbol lprocfs_stats_header (err 0) lustre: Unknown symbol cl_page_print (err 0) lustre: Unknown symbol cl_object_attr_lock (err 0) lustre: Unknown symbol cl_object_kill (err 0) lustre: Unknown symbol lprocfs_stats_free (err 0) lustre: Unknown symbol RMF_NIOBUF_INLINE (err 0) lustre: Unknown symbol server_fill_super (err 0) lustre: Unknown symbol lustre_start_mgc (err 0) lustre: Unknown symbol cl_2queue_init (err 0) lustre: Unknown symbol cl_env_alloc (err 0) lustre: Unknown symbol obd_ioctl_getdata (err 0) lustre: Unknown symbol cl_page_list_add (err 0) lustre: Unknown symbol lu_context_key_get (err 0) lustre: Unknown symbol lu_device_type_init (err 0) lustre: Unknown symbol ldlm_cli_convert (err 0) lustre: Unknown symbol ptlrpc_request_addref (err 0) lustre: Unknown symbol cl_page_clip (err 0) lustre: Unknown symbol lprocfs_wr_root_squash (err 0) lustre: Unknown symbol LNetGetId (err 0) lustre: Unknown symbol lu_device_type_fini (err 0) lustre: Unknown symbol cl_type_setup (err 0) lustre: Unknown symbol obd_heat_get (err 0) lustre: Unknown symbol ldlm_flock_completion_ast (err 0) lustre: Unknown symbol linkea_init_with_rec (err 0) lustre: Unknown symbol lprocfs_counter_add (err 0) lustre: Unknown symbol cl_vmpage_page (err 0) lustre: Unknown symbol ldlm_lockname (err 0) lustre: Unknown symbol cl_site_fini (err 0) lustre: Unknown symbol linkea_entry_unpack (err 0) lustre: Unknown symbol cl_2queue_discard (err 0) lustre: Unknown symbol ldebugfs_stats_seq_fops (err 0) lustre: Unknown symbol lustre_swab_lov_user_md (err 0) lustre: Unknown symbol cl_io_rw_init (err 0) lustre: Unknown symbol cl_object_attr_update (err 0) lustre: Unknown symbol obd_heat_add (err 0) lustre: Unknown symbol sysfs_memparse (err 0) lustre: Unknown symbol lu_object_fini (err 0) lustre: Unknown symbol lustre_sysfs_ops (err 0) lustre: Unknown symbol ldlm_completion_ast (err 0) lustre: Unknown symbol range_lock_tree_init (err 0) lustre: Unknown symbol lprocfs_find_named_value (err 0) lustre: Unknown symbol lmv_stripe_object_get (err 0) lustre: Unknown symbol lu_buf_free (err 0) lustre: Unknown symbol class_put_profile (err 0) lustre: Unknown symbol cl_page_find (err 0) lustre: Unknown symbol lmv_stripe_object_alloc (err 0) lustre: Unknown symbol obdo_from_inode (err 0) lustre: Unknown symbol cl_page_delete (err 0) lustre: Unknown symbol cl_lock_release (err 0) lustre: Unknown symbol cl_sub_dio_free (err 0) lustre: Unknown symbol cl_object_layout_get (err 0) lustre: Unknown symbol cl_sync_io_note (err 0) lustre: Unknown symbol cl_page_own (err 0) lustre: Unknown symbol obd_memory (err 0) lustre: Unknown symbol cl_io_loop (err 0) lustre: Unknown symbol __ldlm_handle2lock (err 0) lustre: Unknown symbol lustre_process_log (err 0) lustre: Unknown symbol lustre_get_jobid (err 0) lustre: Unknown symbol RMF_EAVALS_LENS (err 0) lustre: Unknown symbol obdo_set_o_projid (err 0) lustre: Unknown symbol class_get_profile (err 0) lustre: Unknown symbol lu_site_wq_from_fid (err 0) lustre: Unknown symbol cl_page_header_print (err 0) lustre: Unknown symbol cl_page_is_owned (err 0) lustre: Unknown symbol cfs_free_nidlist (err 0) lustre: Unknown symbol cl_sync_io_wait_recycle (err 0) lustre: Unknown symbol cl_object_put (err 0) lustre: Unknown symbol debugfs_lustre_root (err 0) lustre: Unknown symbol obdo_set_parent_fid (err 0) lustre: Unknown symbol cl_page_touch (err 0) lustre: Unknown symbol lu_object_locate (err 0) lustre: Unknown symbol lu_object_add_top (err 0) lustre: Unknown symbol cl_object_getstripe (err 0) lustre: Unknown symbol ll_dio_user_copy (err 0) lustre: Unknown symbol obd_zombie_barrier (err 0) lustre: Unknown symbol lu_cdebug_printer (err 0) lustre: Unknown symbol ldlm_cli_cancel (err 0) lustre: Unknown symbol lu_context_key_degister_many (err 0) lustre: Unknown symbol cl_page_list_disown (err 0) lustre: Unknown symbol lov_read_and_clear_async_rc (err 0) lustre: Unknown symbol cl_page_slice_add (err 0) lustre: Unknown symbol cl_env_put (err 0) lustre: Unknown symbol lu_device_get (err 0) lustre: Unknown symbol it_open_error (err 0) lustre: Unknown symbol cl_object_attr_unlock (err 0) lustre: Unknown symbol ldlm_lock_put (err 0) lustre: Unknown symbol cl_2queue_fini (err 0) lustre: Unknown symbol lu_context_key_register_many (err 0) lustre: Unknown symbol RMF_FILE_ENCCTX (err 0) lustre: Unknown symbol cl_page_put (err 0) lustre: Unknown symbol cfs_print_nidlist (err 0) lustre: Unknown symbol cl_sync_io_wait (err 0) lustre: Unknown symbol lu_device_init (err 0) lustre: Unknown symbol range_lock (err 0) lustre: Unknown symbol cl_conf_set (err 0) lustre: Unknown symbol cl_sub_dio_alloc (err 0) lustre: Unknown symbol lu_device_fini (err 0) lustre: Unknown symbol lu_site_init_finish (err 0) lustre: Unknown symbol ldlm_lock_allow_match (err 0) lustre: Unknown symbol cl_cache_decref (err 0) lustre: Unknown symbol req_capsule_server_get (err 0) lustre: Unknown symbol ll_release_user_pages (err 0) lustre: Unknown symbol ldlm_lock_decref_and_cancel (err 0) lustre: Unknown symbol lprocfs_stats_collector (err 0) lustre: Unknown symbol unlock_res_and_lock (err 0) lustre: Unknown symbol cl_io_fini (err 0) lustre: Unknown symbol cl_object_fiemap (err 0) lustre: Unknown symbol lock_res_and_lock (err 0) lustre: Unknown symbol cl_site_stats_print (err 0) lustre: Unknown symbol cl_io_top (err 0) lustre: Unknown symbol lu_context_key_revive_many (err 0) lustre: Unknown symbol cl_page_flush (err 0) lustre: Unknown symbol cl_page_unassume (err 0) lustre: Unknown symbol lu_object_get_first (err 0) lustre: Unknown symbol obd_devs (err 0) lustre: Unknown symbol ldebugfs_add_vars (err 0) lustre: Unknown symbol ldlm_lock2handle (err 0) lustre: Unknown symbol cl_io_slice_add (err 0) lustre: Unknown symbol cl_env_percpu_get (err 0) lustre: Unknown symbol cl_page_discard (err 0) lustre: Unknown symbol lu_object_init (err 0) lustre: Unknown symbol cl_lock_request (err 0) lustre: Unknown symbol cl_env_get (err 0) lustre: Unknown symbol lprocfs_counter_init (err 0) lustre: Unknown symbol req_capsule_server_sized_get (err 0) lustre: Unknown symbol cl_page_list_splice (err 0) lustre: Unknown symbol RMF_MDT_BODY (err 0) lustre: Unknown symbol cl_page_disown (err 0) lustre: Unknown symbol cl_object_header_init (err 0) lustre: Unknown symbol RMF_FILE_SECCTX (err 0) lustre: Unknown symbol cl_page_list_init (err 0) lustre: Unknown symbol class_del_profile (err 0) lustre: Unknown symbol cl_stack_fini (err 0) lustre: Unknown symbol lu_kmem_fini (err 0) lustre: Unknown symbol cl_object_prune (err 0) lustre: Unknown symbol class_name2obd (err 0) lustre: Unknown symbol cl_env_percpu_put (err 0) lustre: Unknown symbol cl_object_find (err 0) lustre: Unknown symbol lustre_end_log (err 0) lustre: Unknown symbol lu_kmem_init (err 0) lustre: Unknown symbol cl_2queue_disown (err 0) lustre: Unknown symbol cl_io_commit_async (err 0) lustre: Unknown symbol cl_object_flush (err 0) lustre: Unknown symbol ldlm_it2str (err 0) lustre: Unknown symbol cl_io_lock_add (err 0) lustre: Unknown symbol cl_io_init (err 0) lustre: Unknown symbol cl_page_list_discard (err 0) lustre: Unknown symbol cl_io_lru_reserve (err 0) lustre: Unknown symbol lprocfs_counter_sub (err 0) lustre: Unknown symbol cl_page_list_del (err 0) lustre: Unknown symbol lprocfs_oh_clear (err 0) lustre: Unknown symbol cl_io_submit_rw (err 0) lustre: Unknown symbol lustre_common_put_super (err 0) lustre: Unknown symbol cl_sync_io_init_notify (err 0) lustre: Unknown symbol cl_io_submit_sync (err 0) lustre: Unknown symbol req_capsule_get_size (err 0) lustre: Unknown symbol lu_object_header_fini (err 0) lustre: Unknown symbol lprocfs_stats_alloc (err 0) lustre: Unknown symbol cl_dio_aio_alloc (err 0) lustre: Unknown symbol ll_allocate_dio_buffer (err 0) lustre: Unknown symbol lu_context_key_quiesce_many (err 0) lustre: Unknown symbol obd_heat_clear (err 0) lustre: Unknown symbol cl_io_lock_alloc_add (err 0) lustre: Unknown symbol linkea_data_new (err 0) lustre: Unknown symbol obd_connect_flags2str (err 0) lustre: Unknown symbol RMF_EAVALS (err 0) lustre: Unknown symbol cl_io_read_ahead (err 0) lustre: Unknown symbol cl_env_cache_purge (err 0) lustre: Unknown symbol cl_io_start (err 0) lustre: Unknown symbol lustre_init_lsi (err 0) lustre: Unknown symbol class_exp2obd (err 0) lustre: Unknown symbol lustre_put_lsi (err 0) lustre: Unknown symbol ptlrpc_req_finished (err 0) lustre: Unknown symbol cfs_match_nid (err 0) lustre: Unknown symbol statfs_unpack (err 0) lustre: Unknown symbol ldlm_lock_decref (err 0) lustre: Unknown symbol cl_dio_aio_free (err 0) lustre: Unknown symbol lmd_parse (err 0) lustre: Unknown symbol class_config_llog_handler (err 0) lustre: Unknown symbol lustre_kset (err 0) lustre: Unknown symbol LU_BUF_NULL (err 0) libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: MGS: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: server umount MGS complete Key type lgssc unregistered LNet: 7966:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:131 to 0x2c0000400:161 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 9105:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748561 with bad export cookie 8853873068629998796 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 13183:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:163 to 0x2c0000400:193 Lustre: Mounted lustre-client Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 14317:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748635 with bad export cookie 11126996749072882215 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 14317:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0002-mdc-ffff88028b0d37e8: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: DEBUG MARKER: sleep 60 sec Lustre: DEBUG MARKER: sleep 10 sec Lustre: *** cfs_fail_loc=50f, val=0*** Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request LustreError: 17790:0:(file.c:5394:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 Lustre: Unmounted lustre-client LustreError: 18221:0:(import.c:355:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) LustreError: 18221:0:(import.c:378:ptlrpc_invalidate_import()) @@@ still on sending list req@ffff88008d63b440 x1783364147246592/t0(0) o250->MGC192.168.123.76@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1700748731 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 18221:0:(import.c:389:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. LustreError: 13595:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802a0eda0c0 x1783364147247360/t0(0) o250->MGC192.168.123.76@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: Evicted from MGS (at 192.168.123.76@tcp) after server handle changed from 0x0 to 0x9a6b0897056a607f Lustre: MGC192.168.123.76@tcp: Connection restored to 192.168.123.76@tcp (at 0@lo) LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages LustreError: 16565:0:(ldlm_lockd.c:779:ldlm_handle_ast_error()) ### client (nid 0@lo) returned error from blocking AST (req@ffff8802a0edc140 x1783364147249024 status -107 rc -107), evict it ns: filter-lustre-OST0000_UUID lock: ffff8802d43e1e40/0x9a6b0897056a601d lrc: 4/0,0 mode: PW/PW res: [0x2c0000400:0xc2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x9a6b0897056a6016 expref: 5 pid: 16573 timeout: 123745 lvb_type: 0 LustreError: 138-a: lustre-OST0000: A client on nid 0@lo was evicted due to a lock blocking callback time out: rc -107 LustreError: 14343:0:(ldlm_lockd.c:261:expired_lock_main()) ### lock callback timer expired after 0s: evicting client at 0@lo ns: filter-lustre-OST0000_UUID lock: ffff8802d43e1e40/0x9a6b0897056a601d lrc: 3/0,0 mode: PW/PW res: [0x2c0000400:0xc2:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x60000400030020 nid: 0@lo remote: 0x9a6b0897056a6016 expref: 6 pid: 16573 timeout: 0 lvb_type: 0 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:195 to 0x2c0000400:225 Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 192.168.123.76@tcp (at 0@lo) Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 14321:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748759 with bad export cookie 11126996749072883839 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 7 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 21374:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.76@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_56_gbd04009 LNet: Added LNI 192.168.123.76@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-74.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 27455:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700748883 with bad export cookie 379402092211023201 LustreError: 166-1: MGC192.168.123.76@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages | Link to test |
| replay-single test 71a: mkdir/rmdir striped dir with 2 mdts recovery | BUG: unable to handle kernel paging request at ffff88023d7202f8 IP: [<ffffffffa1356048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33effc067 PMD 33ee10067 PTE 800000023d720060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk libata floppy serio_raw i2c_core [last unloaded: libcfs] CPU: 7 PID: 7254 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88031be69280 ti: ffff880286394000 task.ti: ffff880286394000 RIP: 0010:[<ffffffffa1356048>] [<ffffffffa1356048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880286397cc8 EFLAGS: 00010246 RAX: ffffffffa13ab820 RBX: ffff88026cae33f8 RCX: ffff88024c5d9708 RDX: 0000000000000001 RSI: ffff88023d7202f0 RDI: ffff880286397cf0 RBP: ffff880286397ce0 R08: ffff8800ae6c5fa8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000007 R12: ffff88023d720000 R13: ffff880286397cf0 R14: ffffffffa13ba2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88023d7202f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b5324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b5760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b7385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03cc8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cd278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b0e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b12b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: fail mds2 mds1 1 times Lustre: Failing over lustre-MDT0001 LustreError: 11-0: lustre-MDT0001-osp-MDT0000: operation dt_index_read to node 0@lo failed: rc = -19 LustreError: Skipped 7 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: Failing over lustre-MDT0000 LustreError: 568:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x2400036c9:0x491:0x0].0x0 (ffff8802f54b3c40) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: server umount lustre-MDT0000 complete LustreError: 166-1: MGC192.168.123.33@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 13223:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff880071f22dc0 x1783332104365952/t0(0) o250->MGC192.168.123.33@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0001: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection restored to 192.168.123.33@tcp (at 0@lo) Lustre: Skipped 48 previous similar messages Lustre: lustre-MDT0001: Recovery over after 0:05, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 8 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:547 to 0x380000403:641 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:559 to 0x340000403:673 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:515 to 0x300000403:609 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:579 to 0x2c0000403:673 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:5867 to 0x2c0000405:5889 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:5803 to 0x300000405:5825 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:5867 to 0x380000405:5889 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:5802 to 0x340000405:5825 Lustre: DEBUG MARKER: centos-31.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid,mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: fail mds1 mds2 2 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 LustreError: 19174:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700722963 with bad export cookie 15963021734038791970 | Link to test |
| replay-single test 111b: DNE: unlink striped dir, fail MDT2 | BUG: unable to handle kernel paging request at ffff88008d8802f8 IP: [<ffffffffa135b048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f994067 PTE 800000008d880060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console pcspkr virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks i2c_core floppy virtio_blk serio_raw libata [last unloaded: libcfs] CPU: 14 PID: 26202 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802bfcc3760 ti: ffff880287c50000 task.ti: ffff880287c50000 RIP: 0010:[<ffffffffa135b048>] [<ffffffffa135b048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880287c53cc8 EFLAGS: 00010246 RAX: ffffffffa13b0820 RBX: ffff8802b7d50008 RCX: ffff8802d695d878 RDX: 0000000000000001 RSI: ffff88008d8802f0 RDI: ffff880287c53cf0 RBP: ffff880287c53ce0 R08: ffff880274fddfa8 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000e R12: ffff88008d880000 R13: ffff880287c53cf0 R14: ffffffffa13bf2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88008d8802f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b7324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b7760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b9385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ce8fe>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cf278>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b2e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b32b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Unmounted lustre-client Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-dual test 26: dbench and tar with mds failover | BUG: unable to handle kernel paging request at ffff8802798c02f8 IP: [<ffffffffa136820d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33ec2e067 PTE 80000002798c0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_console virtio_balloon pcspkr ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk serio_raw i2c_core libata floppy [last unloaded: libcfs] CPU: 4 PID: 4433 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802a7860010 ti: ffff8800a3f80000 task.ti: ffff8800a3f80000 RIP: 0010:[<ffffffffa136820d>] [<ffffffffa136820d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8800a3f83cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802b42dc548 RCX: ffff8802929f5964 RDX: 0000000000000004 RSI: ffff8802798c02f0 RDI: ffff8800a687cc88 RBP: ffff8800a3f83ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802798c0000 R13: ffff8800a3f83cf0 R14: ffffffffa13cc2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802798c02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0380324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0380760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0382385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa039794e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03982b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa037be28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa037c2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1673 to 0x2c0000404:1697 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1671 to 0x300000404:1697 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:1672 to 0x340000405:1697 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:1672 to 0x380000405:1697 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 22 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:254 to 0x340000404:289 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:254 to 0x300000405:289 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:254 to 0x380000404:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:253 to 0x2c0000405:289 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:30 to 0x380000403:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:31 to 0x340000403:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:31 to 0x300000403:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:31 to 0x2c0000403:65 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 2184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802165f0d40 x1783185177692480/t154618822755(154618822755) o101->lustre-MDT0000-mdc-ffff8801e774e678@0@lo:12/10 lens 648/608 e 0 to 0 dl 1700579996 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 2184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 59 previous similar messages Lustre: lustre-MDT0000-lwp-OST0001: Connection restored to 192.168.123.73@tcp (at 0@lo) Lustre: Skipped 218 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:1720 to 0x340000405:1761 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:1720 to 0x380000405:1761 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1717 to 0x300000404:1761 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1721 to 0x2c0000404:1761 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 5 times Lustre: Failing over lustre-MDT0001 LustreError: 11-0: lustre-MDT0001-mdc-ffff8801e774e678: operation mds_readpage to node 0@lo failed: rc = -19 LustreError: Skipped 40 previous similar messages LustreError: 25930:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0000-osp-MDT0001: namespace resource [0x200012511:0xf:0x0].0x0 (ffff880233d0a340) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 15 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 827 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:326 to 0x340000404:353 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:323 to 0x2c0000405:353 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:327 to 0x380000404:353 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:320 to 0x300000405:353 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:119 to 0x300000403:161 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:120 to 0x380000403:161 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:120 to 0x340000403:161 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:119 to 0x2c0000403:161 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 203 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1786 to 0x2c0000404:1825 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:1780 to 0x380000405:1825 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:1782 to 0x340000405:1825 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1783 to 0x300000404:1825 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0001: Recovery over after 0:04, of 4 clients 4 recovered and 0 were evicted. Lustre: Skipped 22 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:384 to 0x2c0000405:417 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:378 to 0x380000404:417 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:388 to 0x300000405:417 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:386 to 0x340000404:417 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_26 fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:462 to 0x340000403:481 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:457 to 0x300000403:481 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:427 to 0x380000403:449 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:498 to 0x2c0000403:513 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_26 fail mds1 10 times Lustre: Failing over lustre-MDT0000 LustreError: 21228:0:(ldlm_resource.c:1125:ldlm_resource_complain()) lustre-MDT0001-osp-MDT0000: namespace resource [0x240004282:0xa:0x0].0x0 (ffff8802f53d7240) refcount nonzero (1) after lock cleanup; forcing cleanup. Lustre: server umount lustre-MDT0000 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:1843 to 0x380000405:1889 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:1846 to 0x300000404:1889 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:1846 to 0x2c0000404:1889 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:1844 to 0x340000405:1889 Lustre: DEBUG MARKER: centos-71.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_26 fail mds2 11 times Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff880248b082f8 IP: [<ffffffffa1364048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33edfb067 PMD 33edb5067 PTE 8000000248b08060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console virtio_balloon pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw libata virtio_blk i2c_core floppy [last unloaded: libcfs] CPU: 1 PID: 28652 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c2ec5c40 ti: ffff8802bea58000 task.ti: ffff8802bea58000 RIP: 0010:[<ffffffffa1364048>] [<ffffffffa1364048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802bea5bcc8 EFLAGS: 00010246 RAX: ffffffffa13b9820 RBX: ffff880291589158 RCX: ffff880287ecf818 RDX: 0000000000000001 RSI: ffff880248b082f0 RDI: ffff8802bea5bcf0 RBP: ffff8802bea5bce0 R08: ffff88024f639548 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000001 R12: ffff880248b08000 R13: ffff8802bea5bcf0 R14: ffffffffa13c82c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff880248b082f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03a1324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03a1760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03a3385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03b894e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03b92b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa039ce28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa039d2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=30877 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 LustreError: 11-0: lustre-MDT0000-mdc-ffff8802299d53d8: operation mds_sync to node 0@lo failed: rc = -19 LustreError: Skipped 17 previous similar messages Lustre: lustre-MDT0000-mdc-ffff8802299d53d8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 58 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 191 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 6 previous similar messages LustreError: 30326:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802367d0040 x1783171834619776/t317827580263(317827580263) o101->lustre-MDT0000-mdc-ffff8802299d53d8@0@lo:12/10 lens 624/608 e 0 to 0 dl 1700568502 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 30326:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.8@tcp (at 0@lo) Lustre: Skipped 45 previous similar messages Lustre: lustre-MDT0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:3950 to 0x2c0000405:3969 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:3977 to 0x300000405:4001 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3945 to 0x380000403:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:3945 to 0x340000405:3969 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete LustreError: 30326:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880287848d40 x1783171834616000/t4294967780(4294967780) o101->lustre-MDT0001-mdc-ffff8802299d53d8@0@lo:12/10 lens 576/608 e 0 to 0 dl 1700568531 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 30326:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 18 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3 to 0x340000403:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4 to 0x300000403:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:3 to 0x380000404:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:4 to 0x2c0000403:65 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:3 to 0x340000404:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:4 to 0x380000405:65 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3 to 0x300000404:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:44 to 0x2c0000404:129 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 30326:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802367d0040 x1783171834619776/t317827580263(317827580263) o101->lustre-MDT0000-mdc-ffff8802299d53d8@0@lo:12/10 lens 624/608 e 0 to 0 dl 1700568586 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:4082 to 0x2c0000405:4097 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:4068 to 0x340000405:4097 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:4075 to 0x380000403:4097 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:4110 to 0x300000405:4129 Lustre: DEBUG MARKER: centos-6.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 | Link to test |
| recovery-small test 110k: FID_QUERY failed during recovery | BUG: unable to handle kernel paging request at ffff8802abbe82f8 IP: [<ffffffffa136120d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33ebfa067 PMD 33ea9c067 PTE 80000002abbe8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon pcspkr virtio_console ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm ata_piix serio_raw drm_panel_orientation_quirks virtio_blk floppy libata i2c_core [last unloaded: libcfs] CPU: 15 PID: 5848 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800af3fc9d0 ti: ffff8802c1898000 task.ti: ffff8802c1898000 RIP: 0010:[<ffffffffa136120d>] [<ffffffffa136120d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802c189bcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802e9764548 RCX: ffff880277493cdb RDX: 000000000000000f RSI: ffff8802abbe82f0 RDI: ffff8802cbbd6df8 RBP: ffff8802c189bce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802abbe8000 R13: ffff8802c189bcf0 R14: ffffffffa13c52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331dc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802abbe82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0391324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0391760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0393385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a894e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03a92b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa038ce28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa038d2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: Unmounted lustre-client Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: *** cfs_fail_loc=1103, val=0*** Lustre: lustre-MDT0001: Aborting client recovery LustreError: 15908:0:(ldlm_lib.c:2934:target_stop_recovery_thread()) lustre-MDT0001: Aborting recovery Lustre: 16104:0:(ldlm_lib.c:2317:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: 16104:0:(ldlm_lib.c:2317:target_recovery_overseer()) Skipped 2 previous similar messages Lustre: 16104:0:(genops.c:1481:class_disconnect_stale_exports()) lustre-MDT0001: disconnect stale client lustre-MDT0002-mdtlov_UUID@<unknown> Lustre: 16104:0:(genops.c:1481:class_disconnect_stale_exports()) Skipped 2 previous similar messages Lustre: lustre-MDT0001: disconnecting 2 stale clients Lustre: lustre-MDT0001-osd: cancel update llog [0x240000400:0x1:0x0] Lustre: lustre-MDT0000-osp-MDT0001: cancel update llog [0x200000401:0x1:0x0] Lustre: lustre-MDT0002-osp-MDT0001: cancel update llog [0x280000401:0x1:0x0] Lustre: lustre-OST0003: deleting orphan objects from 0x380000401:628 to 0x380000401:673 Lustre: lustre-OST0001: deleting orphan objects from 0x300000401:627 to 0x300000401:673 Lustre: lustre-OST0002: deleting orphan objects from 0x340000401:628 to 0x340000401:673 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:633 to 0x2c0000401:673 Lustre: Failing over lustre-MDT0001 | Link to test |
| replay-single test 111f: DNE: unlink striped dir, uncommit on MDT1, fail MDT1/MDT2 | BUG: unable to handle kernel paging request at ffff8800738802f8 IP: [<ffffffffa1369048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fc02067 PMD 33fa65067 PTE 8000000073880060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_console pcspkr i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks serio_raw i2c_core virtio_blk floppy libata [last unloaded: libcfs] CPU: 1 PID: 17265 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802b50ddc40 ti: ffff880281a8c000 task.ti: ffff880281a8c000 RIP: 0010:[<ffffffffa1369048>] [<ffffffffa1369048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff880281a8fcc8 EFLAGS: 00010246 RAX: ffffffffa13be820 RBX: ffff8802bb6a5698 RCX: ffff8802833ff538 RDX: 0000000000000002 RSI: ffff8800738802f0 RDI: ffff880281a8fcf0 RBP: ffff880281a8fce0 R08: ffff880244fb9548 R09: 0000000000000000 R10: ffff880331406fc0 R11: 0000000000000001 R12: ffff880073880000 R13: ffff880281a8fcf0 R14: ffffffffa13cd2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800738802f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03c3324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03c3760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03c5385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03da94e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03db2b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03bee28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03bf2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 25 previous similar messages | Link to test |
| conf-sanity test 74: Test per-device adaptive timeout parameters | BUG: unable to handle kernel paging request at ffff88009b5a02f8 IP: [<ffffffffa135b20d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f926067 PTE 800000009b5a0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon i2c_piix4 virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks virtio_blk i2c_core serio_raw floppy libata [last unloaded: libcfs] CPU: 8 PID: 18857 Comm: kworker/u32:4 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff88023fc98010 ti: ffff88027cc70000 task.ti: ffff88027cc70000 RIP: 0010:[<ffffffffa135b20d>] [<ffffffffa135b20d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88027cc73cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802a74133f8 RCX: ffff8802c90fa5e0 RDX: 0000000000000008 RSI: ffff88009b5a02f0 RDI: ffff8802cc100d58 RBP: ffff88027cc73ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88009b5a0000 R13: ffff88027cc73cf0 R14: ffffffffa13bf2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331c00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88009b5a02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa038e324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa038e760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0390385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03a594e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03a62b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa0389e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa038a2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-24.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-24.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:35 to 0x240000400:97 Lustre: Mounted lustre-client Lustre: fs15246-OST0000: new disk, initializing Lustre: srv-fs15246-OST0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall in log fs15246-MDT0000 Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: fs15246-MDT0000: new disk, initializing Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 Lustre: 18221:0:(client.c:1511:after_reply()) @@@ resending request on EINPROGRESS req@ffff88022a196840 x1782912665012032/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1700318138 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 Lustre: Mounted fs15246-client Lustre: Unmounted fs15246-client Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 18221:0:(fid_request.c:233:seq_client_alloc_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Cannot allocate new meta-sequence: rc = -5 LustreError: 18221:0:(fid_request.c:275:seq_client_get_seq()) cli-cli-fs15246-OST0000-osc-MDT0000: Can't allocate new sequence: rc = -5 LustreError: 18221:0:(osp_precreate.c:477:osp_precreate_rollover_new_seq()) fs15246-OST0000-osc-MDT0000: alloc fid error: rc = -5 Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) | Link to test |
| conf-sanity test 53a: check OSS thread count params | BUG: unable to handle kernel paging request at ffff8802d35d02f8 IP: [<ffffffffa135720d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e95e067 PTE 80000002d35d0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 virtio_balloon virtio_console pcspkr ip_tables rpcsec_gss_krb5 drm_kms_helper ttm drm ata_generic pata_acpi drm_panel_orientation_quirks virtio_blk ata_piix serio_raw floppy i2c_core libata [last unloaded: libcfs] CPU: 14 PID: 7786 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800a82d24f0 ti: ffff8802e9acc000 task.ti: ffff8802e9acc000 RIP: 0010:[<ffffffffa135720d>] [<ffffffffa135720d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802e9acfcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88028f4c8008 RCX: ffff8802bddcf56a RDX: 000000000000000e RSI: ffff8802d35d02f0 RDI: ffff88009fc66d78 RBP: ffff8802e9acfce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802d35d0000 R13: ffff8802e9acfcf0 R14: ffffffffa13bb2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802d35d02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03bc324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03bc760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03be385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03d394e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03d42b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b7e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b82b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 28427:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166269 with bad export cookie 15142024682150579956 LustreError: 28427:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 32150:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 945:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166353 with bad export cookie 4416085160109767171 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 5128:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:67 to 0x2c0000402:97 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 6337:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166441 with bad export cookie 15064344096446415931 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 10257:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:67 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 11417:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166527 with bad export cookie 14440076771071268951 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 15712:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:131 to 0x2c0000402:161 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 16921:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166607 with bad export cookie 6273117011024515121 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 16921:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 21414:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:163 to 0x2c0000402:193 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 22613:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166691 with bad export cookie 7849238211117560252 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 26664:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.111@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.111@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST2710: new disk, initializing Lustre: srv-lustre-OST2710: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:2710:ost Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:2710:ost] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x2c0000402 Lustre: lustre-OST03e8: new disk, initializing Lustre: srv-lustre-OST03e8: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:3e8:ost Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:3e8:ost] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x300000402 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 2500:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166818 with bad export cookie 7922875633432823044 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 2500:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Lustre: 10688:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1700166829/real 1700166829] req@ffff880326a92dc0 x1782754023883008/t0(0) o39->lustre-MDT0002-lwp-OST2710@0@lo:12/10 lens 224/224 e 0 to 1 dl 1700166839 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-OST2710 complete Lustre: 10771:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1700166841/real 1700166841] req@ffff880326a90040 x1782754023883392/t0(0) o39->lustre-MDT0002-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1700166851 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-OST03e8 complete Lustre: DEBUG MARKER: centos-109.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: Skipped 2 previous similar messages Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: Skipped 2 previous similar messages Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-109.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages LustreError: 17376:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700166930 with bad export cookie 7922875633432825494 LustreError: 166-1: MGC192.168.123.111@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages | Link to test |
| conf-sanity test 0: single mount setup | BUG: unable to handle kernel paging request at ffff8802c88b02f8 IP: [<ffffffffa135e20d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33e9f9067 PMD 33e9b4067 PTE 80000002c88b0060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix floppy virtio_blk serio_raw i2c_core libata [last unloaded: libcfs] CPU: 1 PID: 27566 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880329b89280 ti: ffff88008ee94000 task.ti: ffff88008ee94000 RIP: 0010:[<ffffffffa135e20d>] [<ffffffffa135e20d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88008ee97cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff8802bd4a5698 RCX: ffff8800a0c539a0 RDX: 0000000000000001 RSI: ffff8802c88b02f0 RDI: ffff8802bf7666f8 RBP: ffff88008ee97ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8802c88b0000 R13: ffff88008ee97cf0 R14: ffffffffa13c22c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8802c88b02f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa042f324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa042f760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0431385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa044694e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa04472b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa042ae28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa042b2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 21790:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700159800 with bad export cookie 2705884129727033865 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 21790:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 8 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 6 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 25538:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:35 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-MDT0000 complete LustreError: 26686:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700159883 with bad export cookie 4495165739611716348 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 30704:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:67 to 0x2c0000402:97 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 31833:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700159947 with bad export cookie 7254803850185211607 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 3471:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:99 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 4670:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160011 with bad export cookie 13088000976216415446 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 8942:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:131 to 0x2c0000402:161 Lustre: Mounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 10059:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160106 with bad export cookie 14118588701754238901 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 13856:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:163 to 0x2c0000402:193 Lustre: Mounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 14980:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160191 with bad export cookie 7894884230147773970 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: lustre-MDT0002-mdc-ffff8802278dc138: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete Key type lgssc unregistered LNet: 19065:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 20190:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.61@tcp: failed processing log, type 1: rc = -5 LustreError: 20190:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.61@tcp: failed processing log, type 4: rc = -110 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 15c-8: MGC192.168.123.61@tcp: Confguration from log lustre-client failed from MGS -5. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info Lustre: Unmounted lustre-client LustreError: 20637:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -5 Lustre: server umount lustre-OST0000 complete Key type lgssc unregistered LNet: 21169:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 24991:0:(llite_lib.c:1388:ll_fill_super()) wrong.lustre: fsname longer than 8 characters: rc = -36 Lustre: Unmounted wrong.lustre-client LustreError: 24991:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -36 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 22300:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160401 with bad export cookie 17965216473005508737 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 25878:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 26973:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.61@tcp: failed processing log, type 1: rc = -5 LustreError: 26973:0:(mgc_request.c:251:do_config_log_add()) MGC192.168.123.61@tcp: failed processing log, type 4: rc = -110 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:225 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: Mounted lustre-client Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages LustreError: 26999:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160558 with bad export cookie 14583923401511817059 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 30968:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 31384:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 506 sleeping for 10000ms LustreError: 31384:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 506 awake Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:257 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 32131:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160642 with bad export cookie 14839908485015402885 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 3749:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:289 Lustre: Mounted lustre-client Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: *** cfs_fail_loc=135, val=0*** LustreError: 8573:0:(obd_config.c:781:class_setup()) setup lustre-MDT0000 failed (-2) LustreError: 8573:0:(obd_config.c:2021:class_config_llog_handler()) MGC192.168.123.61@tcp: cfg command failed: rc = -2 Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f LustreError: 15c-8: MGC192.168.123.61@tcp: Confguration from log lustre-MDT0000 failed from MGS -2. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info LustreError: 8535:0:(tgt_mount.c:1524:server_start_targets()) failed to start server lustre-MDT0000: -2 LustreError: 8535:0:(tgt_mount.c:2216:server_fill_super()) Unable to start targets: -2 LustreError: 8535:0:(obd_config.c:834:class_cleanup()) Device 5 not setup Lustre: server umount lustre-MDT0000 complete LustreError: 8535:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -2 LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.61@tcp (at 0@lo) Lustre: lustre-MDT0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:321 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 5123:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160718 with bad export cookie 798535830591938247 LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 10049:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.61@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.59_36_ga3ddb0f LNet: Added LNI 192.168.123.61@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-59.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:353 Lustre: Mounted lustre-client Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: *** cfs_fail_loc=a05, val=0*** Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) LustreError: 14771:0:(obd_config.c:781:class_setup()) setup lustre-MDT0000 failed (-9) LustreError: 14771:0:(obd_config.c:2021:class_config_llog_handler()) MGC192.168.123.61@tcp: cfg command failed: rc = -9 Lustre: cmd=cf003 0:lustre-MDT0000 1:lustre-MDT0000_UUID 2:0 3:lustre-MDT0000-mdtlov 4:f LustreError: 15c-8: MGC192.168.123.61@tcp: Confguration from log lustre-MDT0000 failed from MGS -9. Communication error between node & MGS, a bad configuration, or other errors. See syslog for more info LustreError: 14739:0:(tgt_mount.c:1524:server_start_targets()) failed to start server lustre-MDT0000: -9 LustreError: 14739:0:(tgt_mount.c:2216:server_fill_super()) Unable to start targets: -9 LustreError: 14739:0:(obd_config.c:834:class_cleanup()) Device 5 not setup LustreError: 11153:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160794 with bad export cookie 5216304737720630877 Lustre: server umount lustre-MDT0000 complete LustreError: 14739:0:(super25.c:188:lustre_fill_super()) llite: Unable to mount <unknown>: rc = -9 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages LustreError: 10463:0:(client.c:1291:ptlrpc_import_delay_req()) @@@ invalidate in flight req@ffff8802ad0a4e40 x1782747751279104/t0(0) o250->MGC192.168.123.61@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.61@tcp (at 0@lo) Lustre: lustre-MDT0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:195 to 0x2c0000402:385 Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 5 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 11151:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1700160829 with bad export cookie 5216304737720631031 LustreError: 11151:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 1 previous similar message LustreError: 166-1: MGC192.168.123.61@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: Skipped 11 previous similar messages | Link to test |
| replay-single test 70b: dbench 3mdts recovery; 1 clients | BUG: unable to handle kernel paging request at ffff88009b9302f8 IP: [<ffffffffa133f048>] lod_device_free+0x48/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f924067 PTE 800000009b930060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_console i2c_piix4 virtio_balloon ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm drm_panel_orientation_quirks ata_piix libata virtio_blk serio_raw i2c_core floppy [last unloaded: libcfs] CPU: 14 PID: 6430 Comm: kworker/u32:2 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff880236ad24f0 ti: ffff8802c9c80000 task.ti: ffff8802c9c80000 RIP: 0010:[<ffffffffa133f048>] [<ffffffffa133f048>] lod_device_free+0x48/0x2c0 [lod] RSP: 0018:ffff8802c9c83cc8 EFLAGS: 00010246 RAX: ffffffffa1394820 RBX: ffff8802be2f1158 RCX: ffff88027fb0de38 RDX: 0000000000000001 RSI: ffff88009b9302f0 RDI: ffff8802c9c83cf0 RBP: ffff8802c9c83ce0 R08: ffff880324db3528 R09: 0000000000000000 R10: ffff880331406fc0 R11: 000000000000000e R12: ffff88009b930000 R13: ffff8802c9c83cf0 R14: ffffffffa13a32c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331d80000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88009b9302f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa03b7324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa03b7760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa03b9385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa03ce94e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03cf2b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa03b2e28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa03b32b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: Started rundbench load pid=25710 ... Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 1 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-mdc-ffff8802f05da548: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 23 previous similar messages LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0000-mdc-ffff8802f05da548: can't stat MDS #0: rc = -107 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 132 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 9 previous similar messages LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802887b4e40 x1782595396882688/t317827580257(317827580257) o101->lustre-MDT0000-mdc-ffff8802f05da548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1700018744 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.83@tcp (at 0@lo) Lustre: Skipped 55 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3930 to 0x300000404:3969 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:3930 to 0x380000405:3969 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:3969 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:3969 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 2 times Lustre: Failing over lustre-MDT0001 Lustre: lustre-MDT0001-mdc-ffff8802f05da548: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 56 previous similar messages Lustre: server umount lustre-MDT0001 complete LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802208e61c0 x1782595396886528/t4294967784(4294967784) o101->lustre-MDT0001-mdc-ffff8802f05da548@0@lo:12/10 lens 624/608 e 0 to 0 dl 1700018769 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:56 to 0x2c0000405:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:56 to 0x300000405:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:57 to 0x380000404:129 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:56 to 0x340000405:129 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 3 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-MDT0002: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. Lustre: Skipped 7 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4 to 0x300000403:65 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3 to 0x380000403:65 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:4 to 0x340000404:65 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:43 to 0x2c0000404:129 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 4 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0000-mdc-ffff8802f05da548: can't stat MDS #0: rc = -107 LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 3 previous similar messages Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: Skipped 8 previous similar messages Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 3 clients reconnect Lustre: Skipped 8 previous similar messages LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802887b4e40 x1782595396882688/t317827580257(317827580257) o101->lustre-MDT0000-mdc-ffff8802f05da548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1700018822 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 32 previous similar messages Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:3930 to 0x380000405:4001 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3930 to 0x300000404:4001 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:4001 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:4001 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 5 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:285 to 0x340000405:321 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:274 to 0x300000405:289 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:274 to 0x380000404:289 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:286 to 0x2c0000405:321 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 6 times Lustre: Failing over lustre-MDT0002 Lustre: lustre-MDT0002: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4 to 0x300000403:97 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3 to 0x380000403:97 Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:4 to 0x340000404:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:43 to 0x2c0000404:161 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 7 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0000-mdc-ffff8802f05da548: can't stat MDS #0: rc = -107 LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 6 previous similar messages LustreError: 166-1: MGC192.168.123.83@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: Skipped 5 previous similar messages LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802887b4e40 x1782595396882688/t317827580257(317827580257) o101->lustre-MDT0000-mdc-ffff8802f05da548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1700018902 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 21 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3930 to 0x300000404:4033 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:4033 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:3930 to 0x380000405:4033 Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:4033 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 8 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:443 to 0x300000405:481 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:475 to 0x2c0000405:513 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:475 to 0x340000405:513 Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:444 to 0x380000404:481 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec LustreError: 28981:0:(osd_handler.c:694:osd_ro()) lustre-MDT0002: *** setting device osd-zfs read-only *** LustreError: 28981:0:(osd_handler.c:694:osd_ro()) Skipped 8 previous similar messages Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 9 times Lustre: Failing over lustre-MDT0002 Lustre: server umount lustre-MDT0002 complete Lustre: lustre-OST0002: deleting orphan objects from 0x340000404:4 to 0x340000404:129 Lustre: lustre-OST0001: deleting orphan objects from 0x300000403:4 to 0x300000403:129 Lustre: lustre-OST0003: deleting orphan objects from 0x380000403:3 to 0x380000403:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000404:43 to 0x2c0000404:193 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0002-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: test_70b fail mds1 10 times Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) lustre-MDT0000-mdc-ffff8802f05da548: can't stat MDS #0: rc = -107 LustreError: 25845:0:(lmv_obd.c:1337:lmv_statfs()) Skipped 6 previous similar messages LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802887b4e40 x1782595396882688/t317827580257(317827580257) o101->lustre-MDT0000-mdc-ffff8802f05da548@0@lo:12/10 lens 576/608 e 0 to 0 dl 1700018984 ref 2 fl Interpret:RPQU/204/0 rc 301/301 job:'dbench.0' uid:0 gid:0 LustreError: 27232:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 18 previous similar messages Lustre: lustre-OST0002: deleting orphan objects from 0x340000403:3930 to 0x340000403:4065 Lustre: lustre-OST0003: deleting orphan objects from 0x380000405:3930 to 0x380000405:4065 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000403:3934 to 0x2c0000403:4065 Lustre: lustre-OST0001: deleting orphan objects from 0x300000404:3930 to 0x300000404:4065 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds2 REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0001 Lustre: DEBUG MARKER: test_70b fail mds2 11 times Lustre: Failing over lustre-MDT0001 Lustre: server umount lustre-MDT0001 complete Lustre: lustre-OST0003: deleting orphan objects from 0x380000404:625 to 0x380000404:641 Lustre: lustre-OST0002: deleting orphan objects from 0x340000405:656 to 0x340000405:673 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000405:653 to 0x2c0000405:673 Lustre: lustre-OST0001: deleting orphan objects from 0x300000405:627 to 0x300000405:673 Lustre: DEBUG MARKER: centos-81.localnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mds3 REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0002 Lustre: DEBUG MARKER: test_70b fail mds3 12 times Lustre: Failing over lustre-MDT0002 | Link to test |
| conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added | BUG: unable to handle kernel paging request at ffff88008fbd82f8 IP: [<ffffffffa136120d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f983067 PTE 800000008fbd8060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common virtio_balloon virtio_console pcspkr i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic pata_acpi ttm drm ata_piix drm_panel_orientation_quirks virtio_blk floppy serio_raw i2c_core libata [last unloaded: libcfs] CPU: 5 PID: 5125 Comm: kworker/u32:3 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802c0d949d0 ti: ffff88009dfc4000 task.ti: ffff88009dfc4000 RIP: 0010:[<ffffffffa136120d>] [<ffffffffa136120d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88009dfc7cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff880275629158 RCX: ffff8802f51b8c65 RDX: 0000000000000005 RSI: ffff88008fbd82f0 RDI: ffff880099903a68 RBP: ffff88009dfc7ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff88008fbd8000 R13: ffff88009dfc7cf0 R14: ffffffffa13c52c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331b40000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff88008fbd82f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa0383324>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa0383760>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0385385>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa039a94e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa039b2b8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa037ee28>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa037f2b2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: 10828:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699874404/real 1699874404] req@ffff8802b2d720c0 x1782447403495552/t0(0) o400->lustre-MDT0002-lwp-OST0001@0@lo:12/10 lens 224/224 e 0 to 1 dl 1699874420 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0002-lwp-OST0001: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-OST0001 complete Lustre: DEBUG MARKER: centos-69.localnet: executing set_hostid Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 4 previous similar messages Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) Lustre: Skipped 5 previous similar messages Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: Skipped 2 previous similar messages Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client LustreError: 24451:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms Lustre: lustre-OST0001: new disk, initializing Lustre: srv-lustre-OST0001: No data found on store. Initialize space: rc = -61 Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid LustreError: 24451:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake LustreError: 24451:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 sleeping for 10000ms Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:1:ost Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x0000000300000400-0x0000000340000400]:1:ost] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec LustreError: 24451:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 142 awake Lustre: lustre-OST0001-osc-MDT0002: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0001 complete LustreError: 137-5: lustre-OST0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Skipped 2 previous similar messages Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 24336:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699874552 with bad export cookie 11151465290099818130 LustreError: 24336:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 8 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 13 previous similar messages | Link to test |
| conf-sanity test 90a: check max_mod_rpcs_in_flight is enforced | BUG: unable to handle kernel paging request at ffff8800989982f8 IP: [<ffffffffa135920d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f93c067 PTE 8000000098998060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common i2c_piix4 pcspkr virtio_balloon virtio_console ip_tables rpcsec_gss_krb5 ata_generic pata_acpi drm_kms_helper ttm drm ata_piix drm_panel_orientation_quirks floppy serio_raw i2c_core virtio_blk libata [last unloaded: libcfs] CPU: 0 PID: 2897 Comm: kworker/u32:0 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8800a020b760 ti: ffff880271528000 task.ti: ffff880271528000 RIP: 0010:[<ffffffffa135920d>] [<ffffffffa135920d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff88027152bcc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88006ee90008 RCX: ffff88008f4b2e6e RDX: 0000000000000000 RSI: ffff8800989982f0 RDI: ffff880071299bf8 RBP: ffff88027152bce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff880098998000 R13: ffff88027152bcf0 R14: ffffffffa13bd2c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331a00000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800989982f8 CR3: 0000000001c10000 CR4: 00000000000007f0 Call Trace: [<ffffffffa060f334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa060f770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0611395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa062695e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa06272c8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa060ae38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa060b2c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Skipped 1 previous similar message Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 5310:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752612/real 1699752612] req@ffff8802675c8d40 x1782318438885504/t0(0) o36->lustre-MDT0000-mdc-ffff8802a1c64138@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752628 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Client 2ec89889-0dfa-4d01-bbf5-c8f2f07577a7 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8802a1c64138: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 5 previous similar messages Lustre: 5364:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752628/real 1699752628] req@ffff8802704a3ac0 x1782318438894592/t0(0) o36->lustre-MDT0000-mdc-ffff8802a1c64138@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752644 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000: Client 2ec89889-0dfa-4d01-bbf5-c8f2f07577a7 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8802a1c64138: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Unmounted lustre-client Lustre: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete Lustre: 5631:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752649/real 1699752649] req@ffff880278b1db40 x1782318438906944/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699752659 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete Lustre: 5854:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752661/real 1699752661] req@ffff8802675c8040 x1782318438909184/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699752671 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 6378:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:42 to 0x2c0000402:65 Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 10493:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752730/real 1699752730] req@ffff88026b1fa0c0 x1782319880884800/t0(0) o36->lustre-MDT0000-mdc-ffff880325afb7e8@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752746 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000-mdc-ffff880325afb7e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Client a87a3512-3bc7-477b-9c2f-8b207b716e37 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff880325afb7e8: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 10693:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752747/real 1699752747] req@ffff88009ac3e1c0 x1782319880898176/t0(0) o36->lustre-MDT0001-mdc-ffff880325afb7e8@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752763 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0001-mdc-ffff880325afb7e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Client a87a3512-3bc7-477b-9c2f-8b207b716e37 (at 0@lo) reconnecting Lustre: lustre-MDT0001-mdc-ffff880325afb7e8: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 3 previous similar messages Lustre: 10883:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752763/real 1699752763] req@ffff88021f0f6ec0 x1782319880907072/t0(0) o36->lustre-MDT0001-mdc-ffff880325afb7e8@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752779 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0001-mdc-ffff880325afb7e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0001: Client a87a3512-3bc7-477b-9c2f-8b207b716e37 (at 0@lo) reconnecting Lustre: lustre-MDT0001-mdc-ffff880325afb7e8: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 4 previous similar messages Lustre: 11168:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752782/real 1699752782] req@ffff880251dd7540 x1782319880931584/t0(0) o36->lustre-MDT0002-mdc-ffff880233bc4138@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752798 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 11168:0:(client.c:2337:ptlrpc_expire_one_request()) Skipped 3 previous similar messages Lustre: lustre-MDT0002-mdc-ffff880233bc4138: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0002: Client 4c9b1e1c-477b-492c-b046-2b069fd21b24 (at 0@lo) reconnecting Lustre: lustre-MDT0002-mdc-ffff880233bc4138: Connection restored to (at 0@lo) Lustre: *** cfs_fail_loc=159, val=0*** Lustre: Skipped 14 previous similar messages Lustre: 11436:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752798/real 1699752798] req@ffff88023f664e40 x1782319880944768/t0(0) o36->lustre-MDT0002-mdc-ffff880233bc4138@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699752814 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0002-mdc-ffff880233bc4138: Connection to lustre-MDT0002 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0002: Client 4c9b1e1c-477b-492c-b046-2b069fd21b24 (at 0@lo) reconnecting Lustre: lustre-MDT0002-mdc-ffff880233bc4138: Connection restored to (at 0@lo) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 7516:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699752827 with bad export cookie 17158690519797324970 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 11-0: lustre-MDT0001-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message LustreError: Skipped 2 previous similar messages Lustre: 12067:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752827/real 1699752827] req@ffff88024bf16840 x1782319880972032/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699752837 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0001 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 12586:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:83 to 0x2c0000400:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:68 to 0x2c0000402:97 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:72 to 0x2c0000401:97 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client Lustre: Mounted lustre-client LustreError: 16821:0:(genops.c:2089:obd_set_max_mod_rpcs_in_flight()) lustre-MDT0000-mdc-ffff8800ab598008: can't set max_mod_rpcs_in_flight=9 higher than mdt.*.max_mod_rpcs_in_flight=8 returned by the MDT server at connection. Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params Lustre: Unmounted lustre-client Lustre: Mounted lustre-client Lustre: Modifying parameter lustre.mdc.lustre-MDT0000-mdc-*.max_rpcs_in_flight in log params Lustre: Skipped 1 previous similar message Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: 17208:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752913/real 1699752913] req@ffff88029b948040 x1782320053913536/t0(0) o9->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699752923 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: server umount lustre-MDT0000 complete LustreError: 13699:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699752925 with bad export cookie 14486821188359009990 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages Lustre: 13002:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752913/real 1699752913] req@ffff8802a07ef540 x1782320053911936/t0(0) o400->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699752929 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 13009:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752914/real 1699752914] req@ffff88029b94db40 x1782320053913792/t0(0) o13->lustre-OST0000-osc-MDT0002@0@lo:7/4 lens 224/368 e 0 to 1 dl 1699752930 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: 13002:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752918/real 1699752918] req@ffff8802ec5ea740 x1782320053914560/t0(0) o400->lustre-OST0000-osc-MDT0002@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699752934 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 5 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 18007:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:72 to 0x2c0000401:129 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:83 to 0x2c0000400:161 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:68 to 0x2c0000402:129 Lustre: Mounted lustre-client Lustre: *** cfs_fail_loc=159, val=0*** Lustre: 22363:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699752997/real 1699752997] req@ffff88022c479a40 x1782320153520384/t0(0) o36->lustre-MDT0000-mdc-ffff8800a41ea548@0@lo:12/10 lens 488/512 e 0 to 1 dl 1699753013 ref 2 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 Lustre: lustre-MDT0000-mdc-ffff8800a41ea548: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Client ccacc087-fa7a-41d7-ad24-eb2b8dfaf143 (at 0@lo) reconnecting Lustre: lustre-MDT0000-mdc-ffff8800a41ea548: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 19134:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699753023 with bad export cookie 2639976619326562870 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: 22837:0:(client.c:2337:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1699753023/real 1699753023] req@ffff8802414cf540 x1782320153536192/t0(0) o9->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1699753033 ref 2 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 23424:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:138 to 0x2c0000402:161 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000401:72 to 0x2c0000401:161 Lustre: Mounted lustre-client Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000400:83 to 0x2c0000400:193 Lustre: setting import lustre-MDT0000_UUID INACTIVE by administrator request Lustre: Unmounted lustre-client Lustre: lustre-MDT0001: evicting 74972e43-6583-47e7-a191-041ff16dab9c (at 0@lo) by administrative request Lustre: Skipped 4 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete LustreError: 167-0: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.123.81@tcp (at 0@lo) LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message LustreError: 167-0: lustre-OST0000-osc-MDT0002: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0002: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: lustre-MDT0000-lwp-MDT0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 5 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 5 previous similar messages Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 6 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages Lustre: server umount lustre-MDT0000 complete LustreError: 24566:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699753133 with bad export cookie 2435964578251941526 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 29160:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 LustreError: 4115:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 90e sleeping for 10000ms LustreError: 4115:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 90e awake Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 11 sec Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:2:mdt] Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff880070230008.ost_server_uuid 50 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff880070230008.ost_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 4014:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699753248 with bad export cookie 5457712883064079576 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 3 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 10511:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x240000400:3 to 0x240000400:33 Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 13233:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699753352 with bad export cookie 7836752880800510994 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: lustre-MDT0001-osp-MDT0002: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 4 previous similar messages LustreError: Skipped 1 previous similar message LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 17055:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0000: new disk, initializing Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall in log lustre-MDT0001 Lustre: srv-lustre-MDT0001: No data found on store. Initialize space: rc = -61 Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: new disk, initializing Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Setting parameter lustre-MDT0002.mdt.identity_upcall in log lustre-MDT0002 Lustre: srv-lustre-MDT0002: No data found on store. Initialize space: rc = -61 Lustre: lustre-MDT0002: new disk, initializing Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2:mdt Lustre: cli-ctl-lustre-MDT0002: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2:mdt] Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: new disk, initializing Lustre: srv-lustre-OST0000: No data found on store. Initialize space: rc = -61 Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:0:ost Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:0:ost] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: Mounted lustre-client Lustre: Unmounted lustre-client LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-MDT0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 11-0: lustre-MDT0000-lwp-MDT0001: operation mds_disconnect to node 0@lo failed: rc = -107 LustreError: Skipped 1 previous similar message LustreError: 23861:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699753463 with bad export cookie 6283781362602315267 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 23861:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 2 previous similar messages Lustre: lustre-MDT0000-lwp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 2 previous similar messages Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 30096:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:3 to 0x2c0000402:33 Lustre: Mounted lustre-client Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 3 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 3 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 3 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 10 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 10 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 10 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 10 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 22 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 22 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 22 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 22 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 22 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 36 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 36 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 36 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 36 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 36 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 78 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 78 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 78 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 78 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 78 previous similar messages LustreError: 31392:0:(osp_precreate.c:688:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 31392:0:(osp_precreate.c:1384:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 31392:0:(osp_precreate.c:688:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 31392:0:(osp_precreate.c:1384:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 155 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 155 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 155 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 155 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 156 previous similar messages LustreError: 31392:0:(osp_precreate.c:688:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 31392:0:(osp_precreate.c:1384:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 LustreError: 31392:0:(osp_precreate.c:688:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 31392:0:(osp_precreate.c:1384:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 289 previous similar messages Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting Lustre: Skipped 289 previous similar messages LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. LustreError: Skipped 289 previous similar messages Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: Skipped 289 previous similar messages Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: Skipped 289 previous similar messages LustreError: 31392:0:(osp_precreate.c:688:osp_precreate_send()) lustre-OST0000-osc-MDT0000: can't precreate: rc = -5 LustreError: 31392:0:(osp_precreate.c:1384:osp_precreate_thread()) lustre-OST0000-osc-MDT0000: cannot precreate objects: rc = -5 Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 399 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 2 previous similar messages LustreError: 137-5: lustre-OST0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: server umount lustre-OST0000 complete Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0000 complete LustreError: 31254:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699753971 with bad export cookie 7534281007078456394 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail LustreError: 31254:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 4 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: Skipped 2 previous similar messages LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) Lustre: server umount lustre-MDT0001 complete Lustre: server umount lustre-MDT0002 complete Key type lgssc unregistered LNet: 3999:0:(lib-ptl.c:956:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 192.168.123.81@tcp libcfs: HW NUMA nodes: 1, HW CPU cores: 16, npartitions: 8 alg: No test for adler32 (adler32-zlib) Lustre: Lustre: Build Version: 2.15.58_187_gef313f8 LNet: Added LNI 192.168.123.81@tcp [8/512/0/180] LNet: Accept secure, port 988 Key type lgssc registered Lustre: Echo OBD driver; http://www.lustre.org/ Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' LustreError: 137-5: lustre-MDT0001_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 LustreError: 137-5: lustre-MDT0002_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 1 previous similar message Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 Lustre: lustre-MDT0002: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:50034 to 0x2c0000402:50977 Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x2c0000402 to 0x2c0000bd0 Lustre: Mounted lustre-client Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 192.168.123.81@tcp (at 0@lo) LustreError: 5310:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 sleeping for 20000ms LustreError: 11-0: lustre-OST0000-osc-MDT0002: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: lustre-OST0000-osc-MDT0002: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: server umount lustre-OST0000 complete LustreError: 5310:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 2107 awake Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid LustreError: 167-0: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 192.168.123.81@tcp (at 0@lo) Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec Lustre: DEBUG MARKER: centos-79.localnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0000-osc-ffff88029751a548.ost_server_uuid 50 Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff88029751a548.ost_server_uuid in FULL state after 0 sec Lustre: Unmounted lustre-client Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 2 previous similar messages Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 3 previous similar messages LustreError: 11-0: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 Lustre: server umount lustre-OST0000 complete LustreError: 11-0: lustre-MDT0000-osp-MDT0002: operation mds_statfs to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) Lustre: Skipped 1 previous similar message Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: 9351:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699754099 with bad export cookie 8887880949620579018 LustreError: 166-1: MGC192.168.123.81@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: server umount lustre-MDT0001 complete | Link to test |
| conf-sanity test 84: check recovery_hard_time | BUG: unable to handle kernel paging request at ffff8800a2a202f8 IP: [<ffffffffa135020d>] lod_device_free+0x20d/0x2c0 [lod] PGD 23c1067 PUD 33fa01067 PMD 33f8eb067 PTE 80000000a2a20060 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC Modules linked in: lustre(OE) ofd(OE) osp(OE) lod(OE) ost(OE) mdt(OE) mdd(OE) mgs(OE) osd_zfs(OE) lquota(OE) lfsck(OE) obdecho(OE) mgc(OE) mdc(OE) lov(OE) osc(OE) lmv(OE) fid(OE) fld(OE) ptlrpc_gss(OE) ptlrpc(OE) obdclass(OE) ksocklnd(OE) lnet(OE) libcfs(OE) zfs(PO) zunicode(PO) zlua(PO) zcommon(PO) znvpair(PO) zavl(PO) icp(PO) spl(O) crc32_generic crc_t10dif crct10dif_generic crct10dif_common pcspkr virtio_balloon virtio_console i2c_piix4 ip_tables rpcsec_gss_krb5 drm_kms_helper ata_generic ttm pata_acpi drm drm_panel_orientation_quirks ata_piix floppy libata i2c_core virtio_blk serio_raw [last unloaded: libcfs] CPU: 7 PID: 24935 Comm: kworker/u32:1 Kdump: loaded Tainted: P OE ------------ 3.10.0-7.9-debug #2 Hardware name: Red Hat KVM, BIOS 1.16.0-3.module_el8.7.0+1218+f626c2ff 04/01/2014 Workqueue: obd_zombid obd_zombie_exp_cull [obdclass] task: ffff8802dda11280 ti: ffff8802dd1c4000 task.ti: ffff8802dd1c4000 RIP: 0010:[<ffffffffa135020d>] [<ffffffffa135020d>] lod_device_free+0x20d/0x2c0 [lod] RSP: 0018:ffff8802dd1c7cc8 EFLAGS: 00010246 RAX: 0000000000000001 RBX: ffff88009e9467e8 RCX: ffff88032562a218 RDX: 0000000000000007 RSI: ffff8800a2a202f0 RDI: ffff880323a21f78 RBP: ffff8802dd1c7ce0 R08: 697665645f646f6c R09: 656572665f656369 R10: 20737365636f7250 R11: 0a64657265746e65 R12: ffff8800a2a20000 R13: ffff8802dd1c7cf0 R14: ffffffffa13b42c0 R15: 0000000000000800 FS: 0000000000000000(0000) GS:ffff880331bc0000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: ffff8800a2a202f8 CR3: 0000000001c10000 CR4: 00000000000007e0 Call Trace: [<ffffffffa034f334>] class_free_dev+0x4e4/0x740 [obdclass] [<ffffffffa034f770>] class_export_put+0x1e0/0x2e0 [obdclass] [<ffffffffa0351395>] class_unlink_export+0x125/0x160 [obdclass] [<ffffffffa036695e>] class_decref_free+0x4e/0x90 [obdclass] [<ffffffffa03672c8>] class_decref+0x48/0xf0 [obdclass] [<ffffffffa034ae38>] class_export_destroy+0x128/0x5a0 [obdclass] [<ffffffffa034b2c2>] obd_zombie_exp_cull+0x12/0x70 [obdclass] [<ffffffff810b243d>] process_one_work+0x18d/0x4a0 [<ffffffff810b3176>] worker_thread+0x126/0x3b0 [<ffffffff810b3050>] ? manage_workers.isra.23+0x2a0/0x2a0 [<ffffffff810ba114>] kthread+0xe4/0xf0 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 [<ffffffff817ede5d>] ret_from_fork_nospec_begin+0x7/0x21 [<ffffffff810ba030>] ? kthread_create_on_node+0x140/0x140 | Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall in log lustre-MDT0000 Lustre: Skipped 2 previous similar messages Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space: rc = -61 Lustre: Skipped 14 previous similar messages Lustre: lustre-MDT0000: new disk, initializing Lustre: Skipped 12 previous similar messages Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-60 Lustre: Skipped 12 previous similar messages Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] Lustre: Skipped 5 previous similar messages Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount FULL mdc.lustre-MDT0002-mdc-*.mds_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x2c0000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x300000402 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 3 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: centos-69.localnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid 50 Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0002.ost_server_uuid in FULL state after 0 sec Lustre: Mounted lustre-client LustreError: 21383:0:(osd_handler.c:694:osd_ro()) lustre-MDT0000: *** setting device osd-zfs read-only *** Lustre: DEBUG MARKER: mds1 REPLAY BARRIER on lustre-MDT0000 Lustre: DEBUG MARKER: local REPLAY BARRIER on lustre-MDT0000 Lustre: Failing over lustre-MDT0000 Lustre: server umount lustre-MDT0000 complete LustreError: 137-5: lustre-MDT0000_UUID: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. LustreError: Skipped 46 previous similar messages LustreError: 166-1: MGC192.168.123.71@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail Lustre: Evicted from MGS (at 192.168.123.71@tcp) after server handle changed from 0xad6a52ce2b265ae7 to 0xad6a52ce2b2759f0 Lustre: MGC192.168.123.71@tcp: Connection restored to (at 0@lo) Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 4 clients reconnect Lustre: 22311:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 0 Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 192.168.123.71@tcp (at 0@lo) LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880265040040 x1782131341031424/t4294967305(4294967305) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573873 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) Skipped 9 previous similar messages LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 1 previous similar message LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880252616ec0 x1782131341032384/t4294967311(4294967311) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573873 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 2 previous similar messages LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 1 previous similar message LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 2 previous similar messages Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) Skipped 3 previous similar messages LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8802526154c0 x1782131341033024/t4294967315(4294967315) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573874 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 1 previous similar message LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 3 previous similar messages LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 6 previous similar messages Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) Skipped 6 previous similar messages LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 6 previous similar messages LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff88022d6be1c0 x1782131341034304/t4294967323(4294967323) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573877 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 3 previous similar messages LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 12 previous similar messages Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) Skipped 12 previous similar messages LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 12 previous similar messages LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880099fa7540 x1782131341036544/t4294967337(4294967337) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573881 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 6 previous similar messages LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 23 previous similar messages Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) Skipped 23 previous similar messages LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 23 previous similar messages LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff880092f9db40 x1782131341040384/t4294967361(4294967361) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573889 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 11 previous similar messages LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 awake LustreError: 22415:0:(fail.c:149:__cfs_fail_timeout_set()) Skipped 49 previous similar messages Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) lustre-MDT0000: extended recovery timer reached hard limit: 60, extend: 1 Lustre: 22415:0:(ldlm_lib.c:1991:extend_recovery_timer()) Skipped 49 previous similar messages LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) cfs_fail_timeout id 709 sleeping for 300ms LustreError: 22415:0:(fail.c:138:__cfs_fail_timeout_set()) Skipped 49 previous similar messages LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) @@@ status 301, old was 0 req@ffff8803265b0040 x1782131341048512/t4294967411(4294967411) o101->lustre-MDT0000-mdc-ffff880276638008@0@lo:12/10 lens 592/608 e 0 to 0 dl 1699573906 ref 2 fl Interpret:RQU/204/0 rc 301/301 job:'' uid:0 gid:0 LustreError: 6184:0:(client.c:3287:ptlrpc_replay_interpret()) Skipped 24 previous similar messages Lustre: 22415:0:(ldlm_lib.c:2299:target_recovery_overseer()) lustre-MDT0000 recovery is aborted by hard timeout Lustre: 22415:0:(ldlm_lib.c:2309:target_recovery_overseer()) recovery is aborted, evict exports in recovery Lustre: 22415:0:(genops.c:1481:class_disconnect_stale_exports()) lustre-MDT0000: disconnect stale client 9354d66c-6f11-4336-b302-06df60f00ceb@0@lo Lustre: lustre-MDT0000: disconnecting 1 stale clients LustreError: 22415:0:(tgt_grant.c:257:tgt_grant_sanity_check()) mdt_obd_disconnect: tot_granted 2097152 != fo_tot_granted 4194304 Lustre: 22415:0:(ldlm_lib.c:1823:abort_req_replay_queue()) @@@ aborted: req@ffff8802bb061a40 x1782131341061312/t0(4294967491) o101->9354d66c-6f11-4336-b302-06df60f00ceb@0@lo:222/0 lens 592/0 e 0 to 0 dl 1699573927 ref 1 fl Complete:/204/ffffffff rc 0/-1 job:'' uid:0 gid:0 LustreError: 11-0: lustre-MDT0000-mdc-ffff880276638008: operation ldlm_enqueue to node 0@lo failed: rc = -107 Lustre: lustre-MDT0000: Denying connection for new client 9354d66c-6f11-4336-b302-06df60f00ceb (at 0@lo), waiting for 4 known clients (3 recovered, 0 in progress, and 1 evicted) already passed deadline 0:00 Lustre: lustre-MDT0000-osd: cancel update llog [0x200000400:0x1:0x0] Lustre: lustre-MDT0001-osp-MDT0000: cancel update llog [0x240000401:0x1:0x0] Lustre: lustre-MDT0002-osp-MDT0000: cancel update llog [0x280000401:0x1:0x0] Lustre: lustre-MDT0000: Recovery over after 1:00, of 4 clients 3 recovered and 1 was evicted. Lustre: lustre-MDT0000-osp-MDT0001: Connection restored to 192.168.123.71@tcp (at 0@lo) Lustre: Skipped 3 previous similar messages Lustre: lustre-OST0001: deleting orphan objects from 0x300000402:49 to 0x300000402:545 Lustre: lustre-OST0000: deleting orphan objects from 0x2c0000402:48 to 0x2c0000402:545 LustreError: 167-0: lustre-MDT0000-mdc-ffff880276638008: This client was evicted by lustre-MDT0000; in progress operations using this service will fail. LustreError: 22470:0:(file.c:5373:ll_inode_revalidate_fini()) lustre: revalidate FID [0x200000007:0x1:0x0] error: rc = -5 Lustre: lustre-MDT0000-mdc-ffff880276638008: Connection restored to 192.168.123.71@tcp (at 0@lo) Lustre: Skipped 2 previous similar messages Lustre: Unmounted lustre-client Lustre: server umount lustre-OST0000 complete Lustre: server umount lustre-OST0001 complete Lustre: server umount lustre-MDT0000 complete LustreError: 13369:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1699573946 with bad export cookie 12495891161533602288 LustreError: 13369:0:(ldlm_lockd.c:2590:ldlm_cancel_handler()) Skipped 8 previous similar messages | Link to test |