Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
Limit to a test: (Copy from below "Failing text"): | |
Delete these reports as invalid (real bug in review or some such) | |
Bug or comment: | |
Extra info: |
Failing Test | Full Crash | Messages before crash | Comment |
---|---|---|---|
sanity-hsm test 407: Check for double RESTORE records in llog | LustreError: 343696:0:(lod_dev.c:2162:lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff9898bae8f000 LustreError: 343696:0:(lod_dev.c:2162:lod_device_free()) LBUG CPU: 1 PID: 343696 Comm: umount Kdump: loaded Tainted: G W OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 lbug_with_loc.cold+0x5/0x43 [libcfs] lod_device_free+0x25b/0x260 [lod] class_free_dev+0x5ce/0x700 [obdclass] class_detach+0x290/0x2e0 [obdclass] class_process_config+0x932/0x1ab0 [obdclass] ? class_manual_cleanup+0x160/0x740 [obdclass] class_manual_cleanup+0x201/0x740 [obdclass] lod_obd_disconnect+0xff/0x1d0 [lod] obd_disconnect.isra.0+0xfe/0x220 [mdd] mdd_process_config+0x186/0x450 [mdd] mdt_stack_fini+0x26c/0x6a0 [mdt] mdt_fini+0x305/0x580 [mdt] mdt_device_fini+0x2b/0xc0 [mdt] obd_precleanup+0xdc/0x280 [obdclass] ? class_disconnect_exports+0x131/0x300 [obdclass] class_cleanup+0x2d7/0x600 [obdclass] class_process_config+0x1102/0x1ab0 [obdclass] ? class_manual_cleanup+0x160/0x740 [obdclass] class_manual_cleanup+0x1e5/0x740 [obdclass] server_put_super+0x98f/0xb40 [ptlrpc] generic_shutdown_super+0x74/0xf0 kill_anon_super+0x12/0x40 deactivate_locked_super+0x31/0xb0 cleanup_mnt+0x100/0x160 task_work_run+0x5c/0x90 exit_to_user_mode_loop+0x12b/0x130 exit_to_user_mode_prepare+0xb9/0x100 syscall_exit_to_user_mode+0x12/0x40 do_syscall_64+0x6b/0xf0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? syscall_exit_to_user_mode+0x19/0x40 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? _raw_spin_unlock_irq+0xa/0x30 ? sigprocmask+0xb4/0xe0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? exc_page_fault+0x62/0x150 entry_SYSCALL_64_after_hwframe+0x78/0x80 RIP: 0033:0x7f296fb0e06b | Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007939:0x136:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n mdt.lustre-MDT0000.hsm.actions | awk '/'0x200007939:0x136:0x0'.*action='ARCHIVE'/ {print $13}' | cut -f2 -d= Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_val=5 fail_loc=0x164 LustreError: 89023:0:(mdt_hsm_cdt_client.c:366:mdt_hsm_register_hal()) cfs_fail_timeout id 164 sleeping for 5000ms Lustre: DEBUG MARKER: /usr/sbin/lctl get_param mdt.lustre-MDT0000.hsm.actions Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d /mnt/lustre-mds1 Lustre: Failing over lustre-MDT0000 Lustre: lustre-MDT0000: Not available for connect from 10.240.28.51@tcp (stopping) Lustre: Skipped 4 previous similar messages LustreError: 89023:0:(mdt_hsm_cdt_client.c:366:mdt_hsm_register_hal()) cfs_fail_timeout id 164 awake LustreError: 343696:0:(obd_class.h:478:obd_check_dev()) Device 18 not setup LustreError: 343696:0:(obd_class.h:478:obd_check_dev()) Skipped 23 previous similar messages LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@0000000028477522[0x4, 1, [0x200000003:0x2:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....local_storage@000000001e12803c LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@0000000016428ebeosd-ldiskfs-object@0000000016428ebe(i:0000000069448979:113/2437543118)[lfix] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@0000000028477522 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@00000000468b74f7[0x5, 2, [0x200007939:0x134:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....mdt@0000000006bc3da1mdt-object@00000000468b74f7( , writecount=0) LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....mdd@000000008b163a92mdd-object@000000008b163a92(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....lod@00000000f370490dlod-object@00000000f370490d LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000ce62b7acosd-ldiskfs-object@00000000ce62b7ac(i:0000000026700157:755/2160698940)[plain] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@00000000468b74f7 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@00000000d027f598[0x4, 1, [0x200000003:0x0:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....local_storage@00000000ef0ecab0 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000c09f1eaaosd-ldiskfs-object@00000000c09f1eaa(i:00000000a5896cd6:112/2575750148)[plain] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@00000000d027f598 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@00000000525e7854[0x5, 1, [0x200000001:0x1017:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....local_storage@00000000729328c3 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000005d46f54bosd-ldiskfs-object@000000005d46f54b(i:000000004b4df590:31713/4006864702)[plain] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@00000000525e7854 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@0000000021812330[0x4, 1, [0xa:0x0:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....local_storage@0000000099d5c007 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000412f21e2osd-ldiskfs-object@00000000412f21e2(i:000000001cc0e736:181/1964332543)[plain] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@0000000021812330 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@000000005580f91a[0x4, 1, [0x200000003:0x3:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....local_storage@000000002319accc LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000c769db83osd-ldiskfs-object@00000000c769db83(i:00000000a19005d6:31715/4127773890)[plain] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@000000005580f91a LustreError: 343696:0:(lu_object.c:984:lu_site_print()) header@00000000078b6a3d[0x4, 1, [0x1:0x0:0x0] hash exist]{ LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....local_storage@0000000055095c30 LustreError: 343696:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000ee6d38e9osd-ldiskfs-object@00000000ee6d38e9(i:000000004f2381a3:147/1840752642)[plain] LustreError: 343696:0:(lu_object.c:984:lu_site_print()) } header@00000000078b6a3d | Link to test |
sanity-lfsck test 15d: LFSCK don't crash upon dir migration failure | LustreError: 121961:0:(lod_dev.c:2162:lod_device_free()) ASSERTION( atomic_read(&lu->ld_ref) == 0 ) failed: lu is ffff97066c87a000 LustreError: 121961:0:(lod_dev.c:2162:lod_device_free()) LBUG CPU: 0 PID: 121961 Comm: umount Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1_lustre.el9.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 lbug_with_loc.cold+0x5/0x43 [libcfs] lod_device_free+0x25b/0x260 [lod] class_free_dev+0x5ce/0x700 [obdclass] class_detach+0x290/0x2e0 [obdclass] class_process_config+0x932/0x1ab0 [obdclass] ? class_manual_cleanup+0x160/0x740 [obdclass] class_manual_cleanup+0x201/0x740 [obdclass] lod_obd_disconnect+0xff/0x1d0 [lod] obd_disconnect.isra.0+0xfe/0x220 [mdd] mdd_process_config+0x186/0x450 [mdd] mdt_stack_fini+0x26c/0x6a0 [mdt] mdt_fini+0x305/0x580 [mdt] mdt_device_fini+0x2b/0xc0 [mdt] obd_precleanup+0xdc/0x280 [obdclass] ? class_disconnect_exports+0x131/0x300 [obdclass] class_cleanup+0x2d7/0x600 [obdclass] class_process_config+0x1102/0x1ab0 [obdclass] ? class_manual_cleanup+0x160/0x740 [obdclass] class_manual_cleanup+0x1e5/0x740 [obdclass] server_put_super+0x98f/0xb40 [ptlrpc] generic_shutdown_super+0x74/0xf0 kill_anon_super+0x12/0x40 deactivate_locked_super+0x31/0xb0 cleanup_mnt+0x100/0x160 task_work_run+0x5c/0x90 exit_to_user_mode_loop+0x12b/0x130 exit_to_user_mode_prepare+0xb9/0x100 syscall_exit_to_user_mode+0x12/0x40 do_syscall_64+0x6b/0xf0 ? _raw_spin_unlock+0xa/0x30 ? flock_lock_inode+0x21c/0x390 ? locks_lock_inode_wait+0x6d/0x180 ? __do_sys_flock+0x134/0x1a0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? do_syscall_64+0x6b/0xf0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? __call_rcu_common.constprop.0+0x117/0x2b0 ? syscall_exit_work+0x103/0x130 ? syscall_exit_to_user_mode+0x19/0x40 ? do_syscall_64+0x6b/0xf0 ? exc_page_fault+0x62/0x150 entry_SYSCALL_64_after_hwframe+0x78/0x80 RIP: 0033:0x7f3fcb70e06b | Lustre: 116276:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 312, rollback = 2 Lustre: 116276:0:(osd_handler.c:1960:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 Lustre: 116276:0:(osd_handler.c:1967:osd_trans_dump_creds()) attr_set: 2/2/0, xattr_set: 6/312/0 Lustre: 116276:0:(osd_handler.c:1974:osd_trans_dump_creds()) write: 3/39/0, punch: 0/0/0, quota 1/3/2 Lustre: 116276:0:(osd_handler.c:1984:osd_trans_dump_creds()) insert: 4/105/1, delete: 0/0/0 Lustre: 116276:0:(osd_handler.c:1991:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 Lustre: 117139:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0002: opcode 2: before 260 < left 387, rollback = 2 Lustre: 117139:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 37 previous similar messages Lustre: 117139:0:(osd_handler.c:1960:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 117139:0:(osd_handler.c:1960:osd_trans_dump_creds()) Skipped 37 previous similar messages Lustre: 117139:0:(osd_handler.c:1967:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/387/0 Lustre: 117139:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 37 previous similar messages Lustre: 117139:0:(osd_handler.c:1974:osd_trans_dump_creds()) write: 5/18/0, punch: 0/0/0, quota 1/3/0 Lustre: 117139:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 37 previous similar messages Lustre: 117139:0:(osd_handler.c:1984:osd_trans_dump_creds()) insert: 1/36/0, delete: 2/2/0 Lustre: 117139:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 37 previous similar messages Lustre: 117139:0:(osd_handler.c:1991:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0 Lustre: 117139:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 37 previous similar messages Lustre: 116276:0:(osd_internal.h:1333:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 387, rollback = 2 Lustre: 116276:0:(osd_internal.h:1333:osd_trans_exec_op()) Skipped 111 previous similar messages Lustre: 116276:0:(osd_handler.c:1960:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 Lustre: 116276:0:(osd_handler.c:1960:osd_trans_dump_creds()) Skipped 111 previous similar messages Lustre: 116276:0:(osd_handler.c:1967:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 5/387/0 Lustre: 116276:0:(osd_handler.c:1967:osd_trans_dump_creds()) Skipped 111 previous similar messages Lustre: 116276:0:(osd_handler.c:1974:osd_trans_dump_creds()) write: 5/18/0, punch: 0/0/0, quota 1/3/0 Lustre: 116276:0:(osd_handler.c:1974:osd_trans_dump_creds()) Skipped 111 previous similar messages Lustre: 116276:0:(osd_handler.c:1984:osd_trans_dump_creds()) insert: 1/36/0, delete: 2/2/0 Lustre: 116276:0:(osd_handler.c:1984:osd_trans_dump_creds()) Skipped 111 previous similar messages Lustre: 116276:0:(osd_handler.c:1991:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 1/1/0 Lustre: 116276:0:(osd_handler.c:1991:osd_trans_dump_creds()) Skipped 111 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1709 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1709 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1709 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lctl lfsck_start -M lustre-MDT0000 -t namespace -A -r Lustre: DEBUG MARKER: grep -c /mnt/lustre-mds1' ' /proc/mounts || true Lustre: DEBUG MARKER: umount -d -f /mnt/lustre-mds1 Lustre: lustre-MDT0000-osp-MDT0002: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 15 previous similar messages LustreError: 121961:0:(obd_class.h:478:obd_check_dev()) Device 14 not setup LustreError: 121961:0:(obd_class.h:478:obd_check_dev()) Skipped 135 previous similar messages LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000001fd6f98a[0x5, 1, [0x200000001:0x1017:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....local_storage@00000000bd58107a LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@0000000022dc144fosd-ldiskfs-object@0000000022dc144f(i:0000000010e22f5c:13/897018906)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000001fd6f98a LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@0000000060860075[0x5, 1, [0x200003ab1:0x1a:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000008dd867bcmdt-object@0000000060860075( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000ae8e0f2bmdd-object@00000000ae8e0f2b(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000d395a6e0lod-object@00000000d395a6e0 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000a46a4467osd-ldiskfs-object@00000000a46a4467(i:0000000072c0706e:193/2988783819)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@0000000060860075 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000b21e771d[0x4, 1, [0x1:0x0:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....local_storage@0000000010b0319e LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000f28091eeosd-ldiskfs-object@00000000f28091ee(i:000000003592a947:83/2000467788)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000b21e771d LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000eec16332[0x5, 1, [0x200003ab1:0x25:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000050b39bemdt-object@00000000eec16332( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000d3370a7fmdd-object@00000000d3370a7f(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000ee32871clod-object@00000000ee32871c LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000e68745fdosd-ldiskfs-object@00000000e68745fd(i:0000000076df8af6:204/3883812860)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000eec16332 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000009b894bd3[0x4, 1, [0xa:0x0:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....local_storage@0000000072d820cb LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000a6ee8452osd-ldiskfs-object@00000000a6ee8452(i:00000000be976661:89/2144999836)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000009b894bd3 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000d5d7039c[0x5, 1, [0x200003ab1:0x1b:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000fd6c7994mdt-object@00000000d5d7039c( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000fd76cda8mdd-object@00000000fd76cda8(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@0000000013adcc76lod-object@0000000013adcc76 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000002847e310osd-ldiskfs-object@000000002847e310(i:000000004297fdb8:194/311866959)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000d5d7039c LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@0000000048897655[0x5, 1, [0x200003ab1:0x14:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000c1f1de9emdt-object@0000000048897655( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@0000000088c0c632mdd-object@0000000088c0c632(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000a149537clod-object@00000000a149537c LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000a6ab9b2aosd-ldiskfs-object@00000000a6ab9b2a(i:000000008dcbce11:187/3525688790)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@0000000048897655 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000005b6eab9e[0x5, 1, [0x200003ab1:0x15:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000009c08bfb8mdt-object@000000005b6eab9e( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000005be17fadmdd-object@000000005be17fad(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000a856dce3lod-object@00000000a856dce3 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000ecdbbc5fosd-ldiskfs-object@00000000ecdbbc5f(i:0000000078a0ea2e:188/3355341755)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000005b6eab9e LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000006cd08781[0x5, 1, [0x200003ab1:0x1c:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@0000000068d9ab5bmdt-object@000000006cd08781( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000007c76147fmdd-object@000000007c76147f(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000d090ba0elod-object@00000000d090ba0e LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000006a4c97b6osd-ldiskfs-object@000000006a4c97b6(i:000000009074bab7:195/346124549)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000006cd08781 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000008fcca689[0x4, 1, [0x200000003:0x3:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....local_storage@0000000041bd9670 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000fa37f26aosd-ldiskfs-object@00000000fa37f26a(i:0000000079ad76f9:20036/1453876007)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000008fcca689 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000008ede8c67[0x5, 1, [0x200003ab1:0x20:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000450a969cmdt-object@000000008ede8c67( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@0000000048c2192emdd-object@0000000048c2192e(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000e3fca2b2lod-object@00000000e3fca2b2 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000003ad8db41osd-ldiskfs-object@000000003ad8db41(i:000000007b8d3fbe:199/2106056651)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000008ede8c67 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000b9fe1413[0x5, 1, [0x200003ab1:0x1d:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000394a92admdt-object@00000000b9fe1413( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000d7d9fae3mdd-object@00000000d7d9fae3(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@000000009c8eba00lod-object@000000009c8eba00 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@0000000005aba9adosd-ldiskfs-object@0000000005aba9ad(i:00000000130b4da2:196/489659719)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000b9fe1413 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000b357a8cb[0x5, 1, [0x200003ab1:0x16:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000004fac2d96mdt-object@00000000b357a8cb( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000b1df90d9mdd-object@00000000b1df90d9(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@000000008ac48c40lod-object@000000008ac48c40 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000005f2da6c1osd-ldiskfs-object@000000005f2da6c1(i:000000007ae61a07:189/1339135021)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000b357a8cb LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000e79c51b4[0x4, 1, [0x200000003:0x2:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....local_storage@00000000b60826df LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000b981b23cosd-ldiskfs-object@00000000b981b23c(i:00000000e87decbd:82/4117020259)[lfix] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000e79c51b4 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000005d42a80a[0x5, 1, [0x200003ab1:0x21:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000009f7864cdmdt-object@000000005d42a80a( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000000ec10c63mdd-object@000000000ec10c63(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@000000009aa5535flod-object@000000009aa5535f LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000b060f563osd-ldiskfs-object@00000000b060f563(i:00000000b8917112:200/3181600228)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000005d42a80a LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000000098399b[0x5, 1, [0x200003ab1:0x17:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000b541685emdt-object@000000000098399b( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000873ac3a7mdd-object@00000000873ac3a7(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@0000000038f754d9lod-object@0000000038f754d9 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000c52b10f5osd-ldiskfs-object@00000000c52b10f5(i:000000007ab0aa05:190/2530109501)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000000098399b LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@0000000097021ded[0x5, 1, [0x200003ab1:0x1e:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@0000000062ddbb6amdt-object@0000000097021ded( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000008a4cb678mdd-object@000000008a4cb678(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@0000000097624d59lod-object@0000000097624d59 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000b8fbf95eosd-ldiskfs-object@00000000b8fbf95e(i:00000000e503ace7:197/1317599331)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@0000000097021ded LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@0000000039c46321[0x5, 1, [0x200003ab1:0x22:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000004757c158mdt-object@0000000039c46321( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000005ee8f1cfmdd-object@000000005ee8f1cf(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000c96e17bdlod-object@00000000c96e17bd LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000b2c44298osd-ldiskfs-object@00000000b2c44298(i:00000000a6252af0:201/1218002066)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@0000000039c46321 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000ccc56576[0x5, 1, [0x200003ab1:0x1f:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000c567f2f2mdt-object@00000000ccc56576( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000d036fe3fmdd-object@00000000d036fe3f(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000d52c8de1lod-object@00000000d52c8de1 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000005a34f68aosd-ldiskfs-object@000000005a34f68a(i:00000000d34525ca:198/2712381000)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000ccc56576 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@0000000092346a92[0x5, 1, [0x200003ab1:0x10:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000009b6ec884mdt-object@0000000092346a92( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000002c5ff2c4mdd-object@000000002c5ff2c4(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000a86cc9e6lod-object@00000000a86cc9e6 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000009a5d0cc8osd-ldiskfs-object@000000009a5d0cc8(i:00000000813d2477:183/196146004)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@0000000092346a92 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000001c8a8039[0x4, 1, [0x200000003:0x0:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....local_storage@00000000faa99040 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@0000000002f5abc4osd-ldiskfs-object@0000000002f5abc4(i:000000009c3a0e8c:81/4013002809)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000001c8a8039 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000d35f2819[0x5, 1, [0x200003ab1:0x23:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000a352b64dmdt-object@00000000d35f2819( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@0000000063e3be7emdd-object@0000000063e3be7e(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000c096c4f4lod-object@00000000c096c4f4 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000002c26fabeosd-ldiskfs-object@000000002c26fabe(i:000000001ecaf206:202/3552828707)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000d35f2819 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000008e49b030[0x5, 1, [0x200003ab1:0x11:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000221aa179mdt-object@000000008e49b030( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@0000000039476fb3mdd-object@0000000039476fb3(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@0000000018c2f8celod-object@0000000018c2f8ce LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000679bab38osd-ldiskfs-object@00000000679bab38(i:00000000d8d2f45f:184/1543775981)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000008e49b030 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000e14ef811[0x5, 1, [0x200003ab1:0x18:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000003fb1d8f8mdt-object@00000000e14ef811( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000779193f7mdd-object@00000000779193f7(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@000000008dd907dblod-object@000000008dd907db LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@000000005d330436osd-ldiskfs-object@000000005d330436(i:00000000310c5d1c:191/1800790897)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000e14ef811 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000002638028b[0x5, 1, [0x200003ab1:0x19:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@000000007205c8e8mdt-object@000000002638028b( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000007f1bb97dmdd-object@000000007f1bb97d(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000ffb7b83dlod-object@00000000ffb7b83d LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000f87433d3osd-ldiskfs-object@00000000f87433d3(i:00000000989d3fb8:192/3190964916)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000002638028b LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000a5d32cba[0x5, 1, [0x200003ab1:0x12:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@0000000054911fb9mdt-object@00000000a5d32cba( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000d069331amdd-object@00000000d069331a(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@0000000014a62ec3lod-object@0000000014a62ec3 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000ed75c8f9osd-ldiskfs-object@00000000ed75c8f9(i:0000000033ffaba4:185/2254121474)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000a5d32cba LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000bd670152[0x5, 1, [0x200003ab1:0xf:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@0000000027572afcmdt-object@00000000bd670152( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@0000000017cad60fmdd-object@0000000017cad60f(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@000000005921cb9clod-object@000000005921cb9c LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000b302fb2dosd-ldiskfs-object@00000000b302fb2d(i:000000008f72ea6a:180/1195990789)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000bd670152 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@00000000d3db3a54[0x5, 1, [0x200003ab1:0x13:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@00000000503cb300mdt-object@00000000d3db3a54( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@00000000e517c701mdd-object@00000000e517c701(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@00000000e2820b93lod-object@00000000e2820b93 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000ed5db903osd-ldiskfs-object@00000000ed5db903(i:00000000e147ed81:186/2559454249)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@00000000d3db3a54 LustreError: 121961:0:(lu_object.c:984:lu_site_print()) header@000000004d84ea3e[0x5, 1, [0x200003ab1:0x24:0x0] hash exist]{ LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdt@0000000069b7d661mdt-object@000000004d84ea3e( , writecount=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....mdd@000000006a9a14dcmdd-object@000000006a9a14dc(open_count=0, valid=0, cltime=0ns, flags=0) LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....lod@0000000097f0914blod-object@0000000097f0914b LustreError: 121961:0:(lu_object.c:984:lu_site_print()) ....osd-ldiskfs@00000000ecea38b9osd-ldiskfs-object@00000000ecea38b9(i:000000007283daf6:203/693433606)[plain] LustreError: 121961:0:(lu_object.c:984:lu_site_print()) } header@000000004d84ea3e | Link to test |