Edit new crash report

Reason	Crashing Function	Where to cut Backtrace	Reports Count
ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed	class_import_put	class_import_put	6

Failing Test	Full Crash	Messages before crash	Comment
recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 8/ iter, 5/30 min)	LustreError: 60019:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed: ------------[ cut here ]------------ LustreError: 60019:0:(genops.c:1240:class_import_put()) LBUG kernel BUG at mm/slub.c:380! invalid opcode: 0000 [#1] SMP NOPTI CPU: 1 PID: 60019 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 CPU: 0 PID: 377 Comm: kworker/u4:5 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: Workqueue: obd_zombid obd_zombie_imp_cull [obdclass] dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] RIP: 0010:__slab_free+0x19b/0x330 class_import_put+0x1d2/0x260 [obdclass]	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); LustreError: 58768:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ff4d9f9c82e58800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 LustreError: 58768:0:(lov_obd.c:792:lov_cleanup()) Skipped 31 previous similar messages Lustre: 58768:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) gssiam: invalidate entry ff4d9f9c8388b000: key 0, ref 0, fl 0, cur 1779435074, ex 629/4199 Lustre: 58768:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) Skipped 8 previous similar messages Lustre: Unmounted lustre-client Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: Mounted lustre-client Lustre: Skipped 3 previous similar messages Lustre: DEBUG MARKER: mount \| grep /mnt/lustre' ' Lustre: lustre-MDT0000-mdc-ff4d9f9c83045800: Connection to lustre-MDT0000 (at 10.240.39.253@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: /usr/bin/lfs df /mnt/lustre > /dev/null 2>&1 \|\| true Lustre: 4276:0:(sec_gssiam.c:387:gssiam_ctx_verify()) lustre-MDT0000-mdc-ff4d9f9c83045800: ctxt from 10.240.39.253@tcp (uid 0) unknown, retrying LustreError: 4276:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -11 req@ff4d9f9c8876ba80 x1865872355668096/t0(0) o38->lustre-MDT0000-mdc-ff4d9f9c83045800@10.240.39.253@tcp:12/10 lens 520/544 e 0 to 0 dl 1779435085 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'lctl.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000-mdc-ff4d9f9c83045800: Connection restored to 10.240.39.253@tcp (at 10.240.39.253@tcp) Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm70.trevis.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: trevis-153vm70.trevis.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff4d9f9c84078000' CPU: 1 PID: 60019 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 sysfs_warn_dup.cold.4+0x17/0x2a sysfs_create_dir_ns+0xb3/0xe0 kobject_add_internal+0xc2/0x290 kobject_init_and_add+0x71/0xa0 ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_init_units+0x72/0x3b0 [obdclass] ? lprocfs_stats_alloc+0x33a/0x680 [obdclass] ldlm_namespace_new+0x759/0xba0 [ptlrpc] client_obd_setup+0x8e3/0x1960 [ptlrpc] osc_setup_common+0xdc/0x2b0 [osc] mdc_setup+0x2c/0x5b0 [mdc] mdc_device_alloc+0xf5/0x240 [mdc] obd_setup+0x220/0x3f0 [obdclass] class_setup+0x483/0x5c0 [obdclass] class_process_config+0x11c3/0x2450 [obdclass] class_config_llog_handler+0x812/0x11b0 [obdclass] llog_process_thread+0xd79/0x1b30 [obdclass] ? llog_validate+0x370/0x370 [obdclass] llog_process_thread_daemonize+0x70/0x90 [obdclass] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 kobject_add_internal failed for lustre-MDT0000-mdc-ff4d9f9c84078000 with -EEXIST, don't try to register things with the same name in the same directory. LustreError: 60019:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff4d9f9c84078000: cannot initialize ns sysfs: rc = -17 LustreError: 60019:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff4d9f9c84078000: unable to create client namespace: rc = -17	Link to test
recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 4/ iter, 2/30 min)	LustreError: 40310:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed: ------------[ cut here ]------------ LustreError: 40310:0:(genops.c:1240:class_import_put()) LBUG kernel BUG at mm/slub.c:380! CPU: 0 PID: 40310 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 invalid opcode: 0000 [#1] SMP NOPTI Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 CPU: 1 PID: 374 Comm: kworker/u4:5 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Call Trace: Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 dump_stack+0x41/0x60 Workqueue: obd_zombid obd_zombie_imp_cull [obdclass] lbug_with_loc.cold.7+0x5/0x43 [libcfs] RIP: 0010:__slab_free+0x19b/0x330 class_import_put+0x1d2/0x260 [obdclass]	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); LustreError: 39065:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ff1a31e9df88c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1 LustreError: 39065:0:(lov_obd.c:792:lov_cleanup()) Skipped 15 previous similar messages Lustre: 39065:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) gssiam: invalidate entry ff1a31e9cdd3bc00: key 0, ref 0, fl 0, cur 1779398924, ex 442/4012 Lustre: 39065:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) Skipped 3 previous similar messages Lustre: Unmounted lustre-client Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: Mounted lustre-client Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: mount \| grep /mnt/lustre' ' Lustre: lustre-MDT0000-mdc-ff1a31e9e83e3000: Connection to lustre-MDT0000 (at 10.240.22.254@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: DEBUG MARKER: /usr/bin/lfs df /mnt/lustre > /dev/null 2>&1 \|\| true Lustre: 4297:0:(sec_gssiam.c:387:gssiam_ctx_verify()) lustre-MDT0000-mdc-ff1a31e9e83e3000: ctxt from 10.240.22.254@tcp (uid 0) unknown, retrying LustreError: 4297:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -11 req@ff1a31e9df610d00 x1865834635091840/t0(0) o38->lustre-MDT0000-mdc-ff1a31e9e83e3000@10.240.22.254@tcp:12/10 lens 520/544 e 0 to 0 dl 1779398934 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'lctl.0' uid:0 gid:0 projid:4294967295 Lustre: lustre-MDT0000-mdc-ff1a31e9e83e3000: Connection restored to 10.240.22.254@tcp (at 10.240.22.254@tcp) Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm241.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-157vm241.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff1a31e9c3911800' CPU: 0 PID: 40310 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 sysfs_warn_dup.cold.4+0x17/0x2a sysfs_create_dir_ns+0xb3/0xe0 kobject_add_internal+0xc2/0x290 kobject_init_and_add+0x71/0xa0 ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_init_units+0x72/0x3b0 [obdclass] ? lprocfs_stats_alloc+0x33a/0x680 [obdclass] ldlm_namespace_new+0x759/0xba0 [ptlrpc] client_obd_setup+0x8e3/0x1960 [ptlrpc] osc_setup_common+0xdc/0x2b0 [osc] mdc_setup+0x2c/0x5b0 [mdc] mdc_device_alloc+0xf5/0x240 [mdc] obd_setup+0x220/0x3f0 [obdclass] class_setup+0x483/0x5c0 [obdclass] class_process_config+0x11c3/0x2450 [obdclass] class_config_llog_handler+0x812/0x11b0 [obdclass] llog_process_thread+0xd79/0x1b30 [obdclass] ? llog_validate+0x370/0x370 [obdclass] llog_process_thread_daemonize+0x70/0x90 [obdclass] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 kobject_add_internal failed for lustre-MDT0000-mdc-ff1a31e9c3911800 with -EEXIST, don't try to register things with the same name in the same directory. LustreError: 40310:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff1a31e9c3911800: cannot initialize ns sysfs: rc = -17 LustreError: 40310:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff1a31e9c3911800: unable to create client namespace: rc = -17	Link to test
recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 10/ iter, 7/30 min)	LustreError: 69988:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed: ------------[ cut here ]------------ LustreError: 69988:0:(genops.c:1240:class_import_put()) LBUG kernel BUG at mm/slub.c:380! invalid opcode: 0000 [#1] SMP NOPTI CPU: 1 PID: 69988 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 CPU: 0 PID: 375 Comm: kworker/u4:3 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: Workqueue: obd_zombid obd_zombie_imp_cull [obdclass] dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] RIP: 0010:__slab_free+0x19b/0x330 class_import_put+0x1d2/0x260 [obdclass]	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: DEBUG MARKER: mount \| grep /mnt/lustre' ' Lustre: lustre-MDT0000-mdc-ff4d1cf1a7470000: Connection to lustre-MDT0000 (at 10.240.23.61@tcp) was lost; in progress operations using this service will wait for recovery to complete Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/bin/lfs df /mnt/lustre > /dev/null 2>&1 \|\| true Lustre: 4341:0:(sec_gssiam.c:386:gssiam_ctx_verify()) lustre-MDT0000-mdc-ff4d1cf1a7470000: ctxt from 10.240.23.61@tcp (uid 0) unknown, retrying Lustre: 4341:0:(sec_gssiam.c:386:gssiam_ctx_verify()) Skipped 1 previous similar message LustreError: 4341:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -11 req@ff4d1cf1a643cd00 x1865786330579456/t0(0) o38->lustre-MDT0000-mdc-ff4d1cf1a7470000@10.240.23.61@tcp:12/10 lens 520/544 e 0 to 0 dl 1779353206 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'lctl.0' uid:0 gid:0 projid:4294967295 LustreError: 4341:0:(client.c:1618:after_reply()) Skipped 1 previous similar message Lustre: lustre-MDT0000-mdc-ff4d1cf1a7470000: Connection restored to 10.240.23.61@tcp (at 10.240.23.61@tcp) Lustre: Skipped 1 previous similar message Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-158vm21.onyx.whamcloud.com: executing wait_import_state_mount \(FULL\\|IDLE\) mdc.lustre-MDT0000-mdc-\.mds_server_uuid Lustre: DEBUG MARKER: onyx-158vm21.onyx.whamcloud.com: executing wait_import_state_mount (FULL\|IDLE) mdc.lustre-MDT0000-mdc-.mds_server_uuid Lustre: DEBUG MARKER: lctl get_param -n at_max Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff4d1cf184745000' CPU: 1 PID: 69988 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 sysfs_warn_dup.cold.4+0x17/0x2a sysfs_create_dir_ns+0xb3/0xe0 kobject_add_internal+0xc2/0x290 kobject_init_and_add+0x71/0xa0 ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_init_units+0x72/0x3b0 [obdclass] ? lprocfs_stats_alloc+0x33a/0x680 [obdclass] ldlm_namespace_new+0x759/0xba0 [ptlrpc] client_obd_setup+0x8e3/0x1960 [ptlrpc] osc_setup_common+0xdc/0x2b0 [osc] mdc_setup+0x2c/0x5b0 [mdc] mdc_device_alloc+0xf5/0x240 [mdc] obd_setup+0x220/0x3f0 [obdclass] class_setup+0x483/0x5c0 [obdclass] class_process_config+0x11c3/0x2450 [obdclass] class_config_llog_handler+0x812/0x11b0 [obdclass] llog_process_thread+0xd79/0x1b30 [obdclass] ? llog_validate+0x370/0x370 [obdclass] llog_process_thread_daemonize+0x70/0x90 [obdclass] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 kobject_add_internal failed for lustre-MDT0000-mdc-ff4d1cf184745000 with -EEXIST, don't try to register things with the same name in the same directory. LustreError: 69988:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff4d1cf184745000: cannot initialize ns sysfs: rc = -17 LustreError: 69988:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff4d1cf184745000: unable to create client namespace: rc = -17	Link to test
recovery-small test 164a: GSSIAM keeps user principal permission after eviction (repeat 29/ iter, 27/30 min)	LustreError: 166204:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed: ------------[ cut here ]------------ LustreError: 166204:0:(genops.c:1240:class_import_put()) LBUG kernel BUG at mm/slub.c:380! CPU: 0 PID: 166204 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 invalid opcode: 0000 [#1] SMP NOPTI Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 CPU: 1 PID: 115438 Comm: kworker/u4:1 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Call Trace: Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 dump_stack+0x41/0x60 Workqueue: obd_zombid obd_zombie_imp_cull [obdclass] lbug_with_loc.cold.7+0x5/0x43 [libcfs] RIP: 0010:__slab_free+0x19b/0x330 class_import_put+0x1d2/0x260 [obdclass]	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all Autotest: Test running for 30 minutes (lustre-reviews_review-dne-zfs-subtest-change_125379.117) Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-*.srpc_info Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: DEBUG MARKER: mount \| grep /mnt/lustre' ' Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff24ecea8d715800' CPU: 0 PID: 166204 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 sysfs_warn_dup.cold.4+0x17/0x2a sysfs_create_dir_ns+0xb3/0xe0 kobject_add_internal+0xc2/0x290 kobject_init_and_add+0x71/0xa0 ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_init_units+0x72/0x3b0 [obdclass] ? lprocfs_stats_alloc+0x33a/0x680 [obdclass] ldlm_namespace_new+0x759/0xba0 [ptlrpc] client_obd_setup+0x8e3/0x1960 [ptlrpc] osc_setup_common+0xdc/0x2b0 [osc] mdc_setup+0x2c/0x5b0 [mdc] mdc_device_alloc+0xf5/0x240 [mdc] obd_setup+0x220/0x3f0 [obdclass] class_setup+0x483/0x5c0 [obdclass] class_process_config+0x11c3/0x2450 [obdclass] class_config_llog_handler+0x812/0x11b0 [obdclass] llog_process_thread+0xd79/0x1b30 [obdclass] ? llog_validate+0x370/0x370 [obdclass] llog_process_thread_daemonize+0x70/0x90 [obdclass] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 kobject_add_internal failed for lustre-MDT0000-mdc-ff24ecea8d715800 with -EEXIST, don't try to register things with the same name in the same directory. LustreError: 166204:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff24ecea8d715800: cannot initialize ns sysfs: rc = -17 LustreError: 166204:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff24ecea8d715800: unable to create client namespace: rc = -17	Link to test
recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 16/ iter, 13/30 min)	LustreError: 98913:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed: ------------[ cut here ]------------ LustreError: 98913:0:(genops.c:1240:class_import_put()) LBUG kernel BUG at mm/slub.c:380! invalid opcode: 0000 [#1] SMP NOPTI CPU: 0 PID: 98913 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 CPU: 1 PID: 43889 Comm: kworker/u4:0 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: Workqueue: obd_zombid obd_zombie_imp_cull [obdclass] dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] RIP: 0010:__slab_free+0x19b/0x330 class_import_put+0x1d2/0x260 [obdclass]	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-*.srpc_info Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); LustreError: 98883:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ff2b5f0b82aab800: lov tgt 0 not cleaned! deathrow=0, lovrc=1 LustreError: 98883:0:(lov_obd.c:792:lov_cleanup()) Skipped 79 previous similar messages Lustre: Unmounted lustre-client Lustre: Skipped 9 previous similar messages Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff2b5f0b9ecf7800' CPU: 0 PID: 98913 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 sysfs_warn_dup.cold.4+0x17/0x2a sysfs_create_dir_ns+0xb3/0xe0 kobject_add_internal+0xc2/0x290 kobject_init_and_add+0x71/0xa0 ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_init_units+0x72/0x3b0 [obdclass] ? lprocfs_stats_alloc+0x33a/0x680 [obdclass] ldlm_namespace_new+0x759/0xba0 [ptlrpc] client_obd_setup+0x8e3/0x1960 [ptlrpc] osc_setup_common+0xdc/0x2b0 [osc] mdc_setup+0x2c/0x5b0 [mdc] mdc_device_alloc+0xf5/0x240 [mdc] obd_setup+0x220/0x3f0 [obdclass] class_setup+0x483/0x5c0 [obdclass] class_process_config+0x11c3/0x2450 [obdclass] class_config_llog_handler+0x812/0x11b0 [obdclass] llog_process_thread+0xd79/0x1b30 [obdclass] ? llog_validate+0x370/0x370 [obdclass] llog_process_thread_daemonize+0x70/0x90 [obdclass] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 kobject_add_internal failed for lustre-MDT0000-mdc-ff2b5f0b9ecf7800 with -EEXIST, don't try to register things with the same name in the same directory. LustreError: 98913:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff2b5f0b9ecf7800: cannot initialize ns sysfs: rc = -17 LustreError: 98913:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff2b5f0b9ecf7800: unable to create client namespace: rc = -17	Link to test
sanity-sec test 306: dynamic gssiam flavor transitions on live mount	LustreError: 774148:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed: ------------[ cut here ]------------ LustreError: 774148:0:(genops.c:1240:class_import_put()) LBUG kernel BUG at mm/slub.c:380! CPU: 1 PID: 774148 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 invalid opcode: 0000 [#1] SMP NOPTI Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 CPU: 0 PID: 605483 Comm: kworker/u4:3 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Call Trace: Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 dump_stack+0x41/0x60 Workqueue: obd_zombid obd_zombie_imp_cull [obdclass] lbug_with_loc.cold.7+0x5/0x43 [libcfs] RIP: 0010:__slab_free+0x19b/0x330 class_import_put+0x1d2/0x260 [obdclass]	Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=null Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=null Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=null Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=null Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: Lustre: DEBUG MARKER: mount \| grep /mnt/lustre' ' Lustre: DEBUG MARKER: /usr/sbin/lctl mark Iteration 1: change flavor to gssiam Lustre: DEBUG MARKER: Iteration 1: change flavor to gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam Lustre: DEBUG MARKER: cat /proc/mounts Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-.srpc_info LustreError: 206216:0:(sec_gssiam.c:377:gssiam_ctx_verify()) lustre-MDT0001-mdc-ff32e2584351e800: GSSIAM init failed: 0/0: rc = -1 LustreError: 206216:0:(sec_gssiam.c:377:gssiam_ctx_verify()) Skipped 1 previous similar message LustreError: 206216:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -1 req@ff32e2588d048340 x1865517717119616/t0(0) o801->lustre-MDT0001-mdc-ff32e2584351e800@10.240.40.225@tcp:12/10 lens 224/224 e 0 to 0 dl 1779098389 ref 2 fl Rpc:RQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 LustreError: 206216:0:(client.c:1618:after_reply()) Skipped 1 previous similar message LustreError: 206216:0:(sec_gssiam.c:669:gssiam_send_init_rpc()) lustre-MDT0001-mdc-ff32e2584351e800: GSSIAM SEC_CTX_INIT failed: rc = -1 LustreError: 206216:0:(sec_gssiam.c:669:gssiam_send_init_rpc()) Skipped 1 previous similar message LustreError: 206216:0:(sec.c:471:sptlrpc_req_get_ctx()) lustre-MDT0001-mdc-ff32e2584351e800: fail to get context for req ff32e2588d049a00: rc = -1 LustreError: 206216:0:(sec.c:471:sptlrpc_req_get_ctx()) Skipped 1 previous similar message LustreError: 206216:0:(pinger.c:128:ptlrpc_ping()) lustre-MDT0001-mdc-ff32e2584351e800: ping failed: rc = -1 Lustre: DEBUG MARKER: lctl get_param -n mdc.--mdc-*.srpc_info Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-sec test_306: @@@@@@ FAIL: Iteration 3: ls failed Lustre: DEBUG MARKER: sanity-sec test_306: @@@@@@ FAIL: Iteration 3: ls failed Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2026-05-18/lustre-reviews_review-dne-zfs-part-2_125211_35_456ef073-25b5-4ac9-99ee-4d49dba6bb44//sanity-sec.test_306.debug_log.$(hostname -s).1779098374.log; Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=super+ioctl+neterror+warning+dlmtrace+error+emerg+ha+rpctrace+vfstrace+config+console+lfsck Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts); LustreError: 774107:0:(lov_obd.c:284:lov_disconnect_obd()) lustre-clilov-ff32e2584351e800: Target lustre-OST0007_UUID disconnect error -1 LustreError: 774107:0:(lmv_obd.c:600:lmv_disconnect_mdc()) lustre-MDT0001-mdc-ff32e2584351e800: Target lustre-MDT0001_UUID disconnect error: rc = -1 Lustre: DEBUG MARKER: mkdir -p /mnt/lustre Lustre: DEBUG MARKER: sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff32e25844348000' CPU: 1 PID: 774148 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 sysfs_warn_dup.cold.4+0x17/0x2a sysfs_create_dir_ns+0xb3/0xe0 kobject_add_internal+0xc2/0x290 kobject_init_and_add+0x71/0xa0 ? srso_alias_return_thunk+0x5/0xfcdfd ? lprocfs_counter_init_units+0x72/0x3b0 [obdclass] ? lprocfs_stats_alloc+0x368/0x680 [obdclass] ldlm_namespace_new+0x759/0xba0 [ptlrpc] client_obd_setup+0x8e3/0x1960 [ptlrpc] ? kobject_uevent_env+0x280/0x660 osc_setup_common+0xdc/0x2b0 [osc] mdc_setup+0x2c/0x5b0 [mdc] mdc_device_alloc+0xf5/0x240 [mdc] obd_setup+0x220/0x3f0 [obdclass] class_setup+0x483/0x5c0 [obdclass] class_process_config+0x11c3/0x2450 [obdclass] ? srso_alias_return_thunk+0x5/0xfcdfd ? srso_alias_return_thunk+0x5/0xfcdfd ? srso_alias_return_thunk+0x5/0xfcdfd ? __kmalloc+0x113/0x250 class_config_llog_handler+0x812/0x11b0 [obdclass] llog_process_thread+0xd79/0x1b30 [obdclass] ? llog_validate+0x370/0x370 [obdclass] llog_process_thread_daemonize+0x70/0x90 [obdclass] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 kobject_add_internal failed for lustre-MDT0000-mdc-ff32e25844348000 with -EEXIST, don't try to register things with the same name in the same directory. LustreError: 774148:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff32e25844348000: cannot initialize ns sysfs: rc = -17 LustreError: 774148:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff32e25844348000: unable to create client namespace: rc = -17	Link to test

Failing Test

Full Crash

Messages before crash

Comment

recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 8/ iter, 5/30 min)

LustreError: 60019:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed:
------------[ cut here ]------------
LustreError: 60019:0:(genops.c:1240:class_import_put()) LBUG
kernel BUG at mm/slub.c:380!
invalid opcode: 0000 [#1] SMP NOPTI
CPU: 1 PID: 60019 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
CPU: 0 PID: 377 Comm: kworker/u4:5 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
Workqueue: obd_zombid obd_zombie_imp_cull [obdclass]
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
RIP: 0010:__slab_free+0x19b/0x330
class_import_put+0x1d2/0x260 [obdclass]

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
LustreError: 58768:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ff4d9f9c82e58800: lov tgt 0 not cleaned! deathrow=0, lovrc=1
LustreError: 58768:0:(lov_obd.c:792:lov_cleanup()) Skipped 31 previous similar messages
Lustre: 58768:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) gssiam: invalidate entry ff4d9f9c8388b000: key 0, ref 0, fl 0, cur 1779435074, ex 629/4199
Lustre: 58768:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) Skipped 8 previous similar messages
Lustre: Unmounted lustre-client
Lustre: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
Lustre: Mounted lustre-client
Lustre: Skipped 3 previous similar messages
Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
Lustre: lustre-MDT0000-mdc-ff4d9f9c83045800: Connection to lustre-MDT0000 (at 10.240.39.253@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: DEBUG MARKER: /usr/bin/lfs df /mnt/lustre > /dev/null 2>&1 || true
Lustre: 4276:0:(sec_gssiam.c:387:gssiam_ctx_verify()) lustre-MDT0000-mdc-ff4d9f9c83045800: ctxt from 10.240.39.253@tcp (uid 0) unknown, retrying
LustreError: 4276:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -11 req@ff4d9f9c8876ba80 x1865872355668096/t0(0) o38->lustre-MDT0000-mdc-ff4d9f9c83045800@10.240.39.253@tcp:12/10 lens 520/544 e 0 to 0 dl 1779435085 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'lctl.0' uid:0 gid:0 projid:4294967295
Lustre: lustre-MDT0000-mdc-ff4d9f9c83045800: Connection restored to 10.240.39.253@tcp (at 10.240.39.253@tcp)
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark trevis-153vm70.trevis.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: trevis-153vm70.trevis.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: lctl get_param -n at_max
Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff4d9f9c84078000'
CPU: 1 PID: 60019 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
sysfs_warn_dup.cold.4+0x17/0x2a
sysfs_create_dir_ns+0xb3/0xe0
kobject_add_internal+0xc2/0x290
kobject_init_and_add+0x71/0xa0
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_init_units+0x72/0x3b0 [obdclass]
? lprocfs_stats_alloc+0x33a/0x680 [obdclass]
ldlm_namespace_new+0x759/0xba0 [ptlrpc]
client_obd_setup+0x8e3/0x1960 [ptlrpc]
osc_setup_common+0xdc/0x2b0 [osc]
mdc_setup+0x2c/0x5b0 [mdc]
mdc_device_alloc+0xf5/0x240 [mdc]
obd_setup+0x220/0x3f0 [obdclass]
class_setup+0x483/0x5c0 [obdclass]
class_process_config+0x11c3/0x2450 [obdclass]
class_config_llog_handler+0x812/0x11b0 [obdclass]
llog_process_thread+0xd79/0x1b30 [obdclass]
? llog_validate+0x370/0x370 [obdclass]
llog_process_thread_daemonize+0x70/0x90 [obdclass]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
kobject_add_internal failed for lustre-MDT0000-mdc-ff4d9f9c84078000 with -EEXIST, don't try to register things with the same name in the same directory.
LustreError: 60019:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff4d9f9c84078000: cannot initialize ns sysfs: rc = -17
LustreError: 60019:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff4d9f9c84078000: unable to create client namespace: rc = -17

Link to test

recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 4/ iter, 2/30 min)

LustreError: 40310:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed:
------------[ cut here ]------------
LustreError: 40310:0:(genops.c:1240:class_import_put()) LBUG
kernel BUG at mm/slub.c:380!
CPU: 0 PID: 40310 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
invalid opcode: 0000 [#1] SMP NOPTI
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
CPU: 1 PID: 374 Comm: kworker/u4:5 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Call Trace:
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
dump_stack+0x41/0x60
Workqueue: obd_zombid obd_zombie_imp_cull [obdclass]
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
RIP: 0010:__slab_free+0x19b/0x330
class_import_put+0x1d2/0x260 [obdclass]

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
LustreError: 39065:0:(lov_obd.c:792:lov_cleanup()) lustre-clilov-ff1a31e9df88c000: lov tgt 0 not cleaned! deathrow=0, lovrc=1
LustreError: 39065:0:(lov_obd.c:792:lov_cleanup()) Skipped 15 previous similar messages
Lustre: 39065:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) gssiam: invalidate entry ff1a31e9cdd3bc00: key 0, ref 0, fl 0, cur 1779398924, ex 442/4012
Lustre: 39065:0:(upcall_cache.c:589:upcall_cache_invalidate_one()) Skipped 3 previous similar messages
Lustre: Unmounted lustre-client
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
Lustre: Mounted lustre-client
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
Lustre: lustre-MDT0000-mdc-ff1a31e9e83e3000: Connection to lustre-MDT0000 (at 10.240.22.254@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: DEBUG MARKER: /usr/bin/lfs df /mnt/lustre > /dev/null 2>&1 || true
Lustre: 4297:0:(sec_gssiam.c:387:gssiam_ctx_verify()) lustre-MDT0000-mdc-ff1a31e9e83e3000: ctxt from 10.240.22.254@tcp (uid 0) unknown, retrying
LustreError: 4297:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -11 req@ff1a31e9df610d00 x1865834635091840/t0(0) o38->lustre-MDT0000-mdc-ff1a31e9e83e3000@10.240.22.254@tcp:12/10 lens 520/544 e 0 to 0 dl 1779398934 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'lctl.0' uid:0 gid:0 projid:4294967295
Lustre: lustre-MDT0000-mdc-ff1a31e9e83e3000: Connection restored to 10.240.22.254@tcp (at 10.240.22.254@tcp)
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm241.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-157vm241.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: lctl get_param -n at_max
Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff1a31e9c3911800'
CPU: 0 PID: 40310 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
sysfs_warn_dup.cold.4+0x17/0x2a
sysfs_create_dir_ns+0xb3/0xe0
kobject_add_internal+0xc2/0x290
kobject_init_and_add+0x71/0xa0
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_init_units+0x72/0x3b0 [obdclass]
? lprocfs_stats_alloc+0x33a/0x680 [obdclass]
ldlm_namespace_new+0x759/0xba0 [ptlrpc]
client_obd_setup+0x8e3/0x1960 [ptlrpc]
osc_setup_common+0xdc/0x2b0 [osc]
mdc_setup+0x2c/0x5b0 [mdc]
mdc_device_alloc+0xf5/0x240 [mdc]
obd_setup+0x220/0x3f0 [obdclass]
class_setup+0x483/0x5c0 [obdclass]
class_process_config+0x11c3/0x2450 [obdclass]
class_config_llog_handler+0x812/0x11b0 [obdclass]
llog_process_thread+0xd79/0x1b30 [obdclass]
? llog_validate+0x370/0x370 [obdclass]
llog_process_thread_daemonize+0x70/0x90 [obdclass]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
kobject_add_internal failed for lustre-MDT0000-mdc-ff1a31e9c3911800 with -EEXIST, don't try to register things with the same name in the same directory.
LustreError: 40310:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff1a31e9c3911800: cannot initialize ns sysfs: rc = -17
LustreError: 40310:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff1a31e9c3911800: unable to create client namespace: rc = -17

Link to test

recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 10/ iter, 7/30 min)

LustreError: 69988:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed:
------------[ cut here ]------------
LustreError: 69988:0:(genops.c:1240:class_import_put()) LBUG
kernel BUG at mm/slub.c:380!
invalid opcode: 0000 [#1] SMP NOPTI
CPU: 1 PID: 69988 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
CPU: 0 PID: 375 Comm: kworker/u4:3 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
Workqueue: obd_zombid obd_zombie_imp_cull [obdclass]
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
RIP: 0010:__slab_free+0x19b/0x330
class_import_put+0x1d2/0x260 [obdclass]

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
Lustre: lustre-MDT0000-mdc-ff4d1cf1a7470000: Connection to lustre-MDT0000 (at 10.240.23.61@tcp) was lost; in progress operations using this service will wait for recovery to complete
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: /usr/bin/lfs df /mnt/lustre > /dev/null 2>&1 || true
Lustre: 4341:0:(sec_gssiam.c:386:gssiam_ctx_verify()) lustre-MDT0000-mdc-ff4d1cf1a7470000: ctxt from 10.240.23.61@tcp (uid 0) unknown, retrying
Lustre: 4341:0:(sec_gssiam.c:386:gssiam_ctx_verify()) Skipped 1 previous similar message
LustreError: 4341:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -11 req@ff4d1cf1a643cd00 x1865786330579456/t0(0) o38->lustre-MDT0000-mdc-ff4d1cf1a7470000@10.240.23.61@tcp:12/10 lens 520/544 e 0 to 0 dl 1779353206 ref 1 fl Rpc:RNQU/200/ffffffff rc 0/-1 job:'lctl.0' uid:0 gid:0 projid:4294967295
LustreError: 4341:0:(client.c:1618:after_reply()) Skipped 1 previous similar message
Lustre: lustre-MDT0000-mdc-ff4d1cf1a7470000: Connection restored to 10.240.23.61@tcp (at 10.240.23.61@tcp)
Lustre: Skipped 1 previous similar message
Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/usr/lib64/lustre/u
Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-158vm21.onyx.whamcloud.com: executing wait_import_state_mount $FULL\|IDLE$ mdc.lustre-MDT0000-mdc-\*.mds_server_uuid
Lustre: DEBUG MARKER: onyx-158vm21.onyx.whamcloud.com: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid
Lustre: DEBUG MARKER: lctl get_param -n at_max
Lustre: DEBUG MARKER: /usr/sbin/lctl mark mdc.lustre-MDT0000-mdc-\*.mds_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff4d1cf184745000'
CPU: 1 PID: 69988 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
sysfs_warn_dup.cold.4+0x17/0x2a
sysfs_create_dir_ns+0xb3/0xe0
kobject_add_internal+0xc2/0x290
kobject_init_and_add+0x71/0xa0
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_init_units+0x72/0x3b0 [obdclass]
? lprocfs_stats_alloc+0x33a/0x680 [obdclass]
ldlm_namespace_new+0x759/0xba0 [ptlrpc]
client_obd_setup+0x8e3/0x1960 [ptlrpc]
osc_setup_common+0xdc/0x2b0 [osc]
mdc_setup+0x2c/0x5b0 [mdc]
mdc_device_alloc+0xf5/0x240 [mdc]
obd_setup+0x220/0x3f0 [obdclass]
class_setup+0x483/0x5c0 [obdclass]
class_process_config+0x11c3/0x2450 [obdclass]
class_config_llog_handler+0x812/0x11b0 [obdclass]
llog_process_thread+0xd79/0x1b30 [obdclass]
? llog_validate+0x370/0x370 [obdclass]
llog_process_thread_daemonize+0x70/0x90 [obdclass]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
kobject_add_internal failed for lustre-MDT0000-mdc-ff4d1cf184745000 with -EEXIST, don't try to register things with the same name in the same directory.
LustreError: 69988:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff4d1cf184745000: cannot initialize ns sysfs: rc = -17
LustreError: 69988:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff4d1cf184745000: unable to create client namespace: rc = -17

Link to test

recovery-small test 164a: GSSIAM keeps user principal permission after eviction (repeat 29/ iter, 27/30 min)

LustreError: 166204:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed:
------------[ cut here ]------------
LustreError: 166204:0:(genops.c:1240:class_import_put()) LBUG
kernel BUG at mm/slub.c:380!
CPU: 0 PID: 166204 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
invalid opcode: 0000 [#1] SMP NOPTI
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
CPU: 1 PID: 115438 Comm: kworker/u4:1 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Call Trace:
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
dump_stack+0x41/0x60
Workqueue: obd_zombid obd_zombie_imp_cull [obdclass]
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
RIP: 0010:__slab_free+0x19b/0x330
class_import_put+0x1d2/0x260 [obdclass]

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all
Autotest: Test running for 30 minutes (lustre-reviews_review-dne-zfs-subtest-change_125379.117)
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff24ecea8d715800'
CPU: 0 PID: 166204 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
sysfs_warn_dup.cold.4+0x17/0x2a
sysfs_create_dir_ns+0xb3/0xe0
kobject_add_internal+0xc2/0x290
kobject_init_and_add+0x71/0xa0
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_init_units+0x72/0x3b0 [obdclass]
? lprocfs_stats_alloc+0x33a/0x680 [obdclass]
ldlm_namespace_new+0x759/0xba0 [ptlrpc]
client_obd_setup+0x8e3/0x1960 [ptlrpc]
osc_setup_common+0xdc/0x2b0 [osc]
mdc_setup+0x2c/0x5b0 [mdc]
mdc_device_alloc+0xf5/0x240 [mdc]
obd_setup+0x220/0x3f0 [obdclass]
class_setup+0x483/0x5c0 [obdclass]
class_process_config+0x11c3/0x2450 [obdclass]
class_config_llog_handler+0x812/0x11b0 [obdclass]
llog_process_thread+0xd79/0x1b30 [obdclass]
? llog_validate+0x370/0x370 [obdclass]
llog_process_thread_daemonize+0x70/0x90 [obdclass]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
kobject_add_internal failed for lustre-MDT0000-mdc-ff24ecea8d715800 with -EEXIST, don't try to register things with the same name in the same directory.
LustreError: 166204:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff24ecea8d715800: cannot initialize ns sysfs: rc = -17
LustreError: 166204:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff24ecea8d715800: unable to create client namespace: rc = -17

Link to test

recovery-small test 164d: GSSIAM keeps permissions after network reconnect without umount (repeat 16/ iter, 13/30 min)

LustreError: 98913:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed:
------------[ cut here ]------------
LustreError: 98913:0:(genops.c:1240:class_import_put()) LBUG
kernel BUG at mm/slub.c:380!
invalid opcode: 0000 [#1] SMP NOPTI
CPU: 0 PID: 98913 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
CPU: 1 PID: 43889 Comm: kworker/u4:0 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
Workqueue: obd_zombid obd_zombie_imp_cull [obdclass]
dump_stack+0x41/0x60
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
RIP: 0010:__slab_free+0x19b/0x330
class_import_put+0x1d2/0x260 [obdclass]

Link to test

sanity-sec test 306: dynamic gssiam flavor transitions on live mount

LustreError: 774148:0:(genops.c:1240:class_import_put()) ASSERTION( refcount_read(&imp->imp_refcount) > 0 ) failed:
------------[ cut here ]------------
LustreError: 774148:0:(genops.c:1240:class_import_put()) LBUG
kernel BUG at mm/slub.c:380!
CPU: 1 PID: 774148 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
invalid opcode: 0000 [#1] SMP NOPTI
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
CPU: 0 PID: 605483 Comm: kworker/u4:3 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Call Trace:
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
dump_stack+0x41/0x60
Workqueue: obd_zombid obd_zombie_imp_cull [obdclass]
lbug_with_loc.cold.7+0x5/0x43 [libcfs]
RIP: 0010:__slab_free+0x19b/0x330
class_import_put+0x1d2/0x260 [obdclass]

Lustre: DEBUG MARKER: /usr/sbin/lctl set_param debug=+all
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0x1205
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param sptlrpc.gssiam.gssiam_upcall=/usr/sbin/l_gssiam_upcall
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=null
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=null
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=null
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=null
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
Lustre: DEBUG MARKER: mount | grep /mnt/lustre' '
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Iteration 1: change flavor to gssiam
Lustre: DEBUG MARKER: Iteration 1: change flavor to gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2mdt=gssiam
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: Setting sptlrpc rule: lustre.srpc.flavor.default.cli2ost=gssiam
Lustre: DEBUG MARKER: cat /proc/mounts
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
LustreError: 206216:0:(sec_gssiam.c:377:gssiam_ctx_verify()) lustre-MDT0001-mdc-ff32e2584351e800: GSSIAM init failed: 0/0: rc = -1
LustreError: 206216:0:(sec_gssiam.c:377:gssiam_ctx_verify()) Skipped 1 previous similar message
LustreError: 206216:0:(client.c:1618:after_reply()) @@@ unwrap reply failed: rc = -1 req@ff32e2588d048340 x1865517717119616/t0(0) o801->lustre-MDT0001-mdc-ff32e2584351e800@10.240.40.225@tcp:12/10 lens 224/224 e 0 to 0 dl 1779098389 ref 2 fl Rpc:RQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295
LustreError: 206216:0:(client.c:1618:after_reply()) Skipped 1 previous similar message
LustreError: 206216:0:(sec_gssiam.c:669:gssiam_send_init_rpc()) lustre-MDT0001-mdc-ff32e2584351e800: GSSIAM SEC_CTX_INIT failed: rc = -1
LustreError: 206216:0:(sec_gssiam.c:669:gssiam_send_init_rpc()) Skipped 1 previous similar message
LustreError: 206216:0:(sec.c:471:sptlrpc_req_get_ctx()) lustre-MDT0001-mdc-ff32e2584351e800: fail to get context for req ff32e2588d049a00: rc = -1
LustreError: 206216:0:(sec.c:471:sptlrpc_req_get_ctx()) Skipped 1 previous similar message
LustreError: 206216:0:(pinger.c:128:ptlrpc_ping()) lustre-MDT0001-mdc-ff32e2584351e800: ping failed: rc = -1
Lustre: DEBUG MARKER: lctl get_param -n mdc.*-*-mdc-*.srpc_info
Lustre: DEBUG MARKER: /usr/sbin/lctl mark sanity-sec test_306: @@@@@@ FAIL: Iteration 3: ls failed
Lustre: DEBUG MARKER: sanity-sec test_306: @@@@@@ FAIL: Iteration 3: ls failed
Lustre: DEBUG MARKER: /usr/sbin/lctl dk > /autotest/autotest-2/2026-05-18/lustre-reviews_review-dne-zfs-part-2_125211_35_456ef073-25b5-4ac9-99ee-4d49dba6bb44//sanity-sec.test_306.debug_log.$(hostname -s).1779098374.log;
Lustre: DEBUG MARKER: /usr/sbin/lctl set_param -n debug=super+ioctl+neterror+warning+dlmtrace+error+emerg+ha+rpctrace+vfstrace+config+console+lfsck
Lustre: DEBUG MARKER: running=$(grep -c /mnt/lustre' ' /proc/mounts);
LustreError: 774107:0:(lov_obd.c:284:lov_disconnect_obd()) lustre-clilov-ff32e2584351e800: Target lustre-OST0007_UUID disconnect error -1
LustreError: 774107:0:(lmv_obd.c:600:lmv_disconnect_mdc()) lustre-MDT0001-mdc-ff32e2584351e800: Target lustre-MDT0001_UUID disconnect error: rc = -1
Lustre: DEBUG MARKER: mkdir -p /mnt/lustre
Lustre: DEBUG MARKER:
sysfs: cannot create duplicate filename '/fs/lustre/ldlm/namespaces/lustre-MDT0000-mdc-ff32e25844348000'
CPU: 1 PID: 774148 Comm: llog_process_th Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.117.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014
Call Trace:
dump_stack+0x41/0x60
sysfs_warn_dup.cold.4+0x17/0x2a
sysfs_create_dir_ns+0xb3/0xe0
kobject_add_internal+0xc2/0x290
kobject_init_and_add+0x71/0xa0
? srso_alias_return_thunk+0x5/0xfcdfd
? lprocfs_counter_init_units+0x72/0x3b0 [obdclass]
? lprocfs_stats_alloc+0x368/0x680 [obdclass]
ldlm_namespace_new+0x759/0xba0 [ptlrpc]
client_obd_setup+0x8e3/0x1960 [ptlrpc]
? kobject_uevent_env+0x280/0x660
osc_setup_common+0xdc/0x2b0 [osc]
mdc_setup+0x2c/0x5b0 [mdc]
mdc_device_alloc+0xf5/0x240 [mdc]
obd_setup+0x220/0x3f0 [obdclass]
class_setup+0x483/0x5c0 [obdclass]
class_process_config+0x11c3/0x2450 [obdclass]
? srso_alias_return_thunk+0x5/0xfcdfd
? srso_alias_return_thunk+0x5/0xfcdfd
? srso_alias_return_thunk+0x5/0xfcdfd
? __kmalloc+0x113/0x250
class_config_llog_handler+0x812/0x11b0 [obdclass]
llog_process_thread+0xd79/0x1b30 [obdclass]
? llog_validate+0x370/0x370 [obdclass]
llog_process_thread_daemonize+0x70/0x90 [obdclass]
kthread+0x134/0x150
? set_kthread_struct+0x50/0x50
ret_from_fork+0x1f/0x40
kobject_add_internal failed for lustre-MDT0000-mdc-ff32e25844348000 with -EEXIST, don't try to register things with the same name in the same directory.
LustreError: 774148:0:(ldlm_resource.c:1038:ldlm_namespace_new()) lustre-MDT0000-mdc-ff32e25844348000: cannot initialize ns sysfs: rc = -17
LustreError: 774148:0:(ldlm_lib.c:599:client_obd_setup()) lustre-MDT0000-mdc-ff32e25844348000: unable to create client namespace: rc = -17

Link to test

Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below):
Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below):
Limit to a test: (Copy from below "Failing text"):
Delete these reports as invalid (real bug in review or some such)
Bug or comment:
Extra info:

Editing crashreport #74915

Added fields:

Failures list (last 100):