LNetError: 1191577:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1191577:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1191577 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __kmem_cache_alloc_node+0x18f/0x2e0
? netlink_realloc_groups+0xbe/0x120
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? syscall_exit_work+0x103/0x130
? do_syscall_64+0x6b/0xf0
? __sys_recvmsg+0x56/0xa0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? handle_mm_fault+0x116/0x270
? do_user_addr_fault+0x1d6/0x6a0
? do_syscall_64+0x6b/0xf0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f86d270fa17

LNet: 1190047:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.232@tcp
LNet: Removed LNI 10.240.25.232@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.232@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.176@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.176@tcp
LNet: There was an unexpected network error while writing to 10.240.28.176: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.28.176@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.28.176@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.176@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-105vm13 to discover 10.240.25.232@tcp
Lustre: DEBUG MARKER: Force onyx-105vm13 to discover 10.240.25.232@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1190920
Lustre: DEBUG MARKER: Wait for 1190920
LNet: There was an unexpected network error while writing to 10.240.28.176: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1190920
Lustre: DEBUG MARKER: Finished wait on 1190920
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1134379:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1134379:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1134379 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __skb_datagram_iter+0x7c/0x2b0
? __pfx_simple_copy_to_iter+0x10/0x10
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? syscall_exit_work+0x103/0x130
? do_syscall_64+0x6b/0xf0
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? __sys_recvmsg+0x56/0xa0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f8070d0fa17

LNet: 1132972:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.239@tcp
LNet: Removed LNI 10.240.25.239@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.239@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.161@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s *@tcp -d *@tcp -r 1 -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.161@tcp
LNet: There was an unexpected network error while writing to 10.240.25.161: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.25.161@tcp -r 1 -m GET -l 3
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s *@tcp -d 10.240.25.161@tcp -r 1 -m GET -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.25.161@tcp -r 1 -m PUT -l 6
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm8.onyx.whamcloud.com to discover 10.240.25.161@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm8.onyx.whamcloud.com to discover 10.240.25.161@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.161@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-66vm10 to discover 10.240.25.239@tcp
Lustre: DEBUG MARKER: Force onyx-66vm10 to discover 10.240.25.239@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1133780
Lustre: DEBUG MARKER: Wait for 1133780
LNet: There was an unexpected network error while writing to 10.240.25.161: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1133780
Lustre: DEBUG MARKER: Finished wait on 1133780
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1036612:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1036612:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1036612 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.38.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_recvmsg+0x212/0x290
? _copy_to_user+0x1a/0x30
? move_addr_to_user+0x4b/0xe0
? ____sys_recvmsg+0xeb/0x1b0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? idr_get_next_ul+0xb6/0xf0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_setsockopt+0x2f7/0x430
? do_sock_setsockopt+0xb7/0x180
? __sys_setsockopt+0x75/0xc0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? do_user_addr_fault+0x1d6/0x6a0
? syscall_exit_work+0x103/0x130
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fbdf9b0fa17

LNet: 1035205:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.22.240@tcp
LNet: Removed LNI 10.240.22.240@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.22.240@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.190@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.190@tcp
LNet: There was an unexpected network error while writing to 10.240.28.190: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.28.190@tcp -r 1 -m GET -l 3
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s *@tcp -d 10.240.28.190@tcp -r 1 -m GET -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.28.190@tcp -r 1 -m PUT -l 6
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-32vm1.onyx.whamcloud.com to discover 10.240.28.190@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-32vm1.onyx.whamcloud.com to discover 10.240.28.190@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.190@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm7 to discover 10.240.22.240@tcp
Lustre: DEBUG MARKER: Force onyx-106vm7 to discover 10.240.22.240@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1036013
LNet: There was an unexpected network error while writing to 10.240.28.190: rc = -22
Lustre: DEBUG MARKER: Wait for 1036013
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1036013
Lustre: DEBUG MARKER: Finished wait on 1036013
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1136674:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1136674:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1136674 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.38.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __check_object_size.part.0+0x35/0xd0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? do_syscall_64+0x6b/0xf0
? __sys_recvmsg+0x56/0xa0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? __check_object_size.part.0+0x35/0xd0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? ___sys_recvmsg+0x88/0xd0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? __sys_recvmsg+0x56/0xa0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f86aab0fa17

LNet: 1135229:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.43.140@tcp
LNet: Removed LNI 10.240.43.140@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.43.140@tcp [8/256/0/180]
LNet: Accept all, port 7988
Autotest: Test running for 255 minutes (lustre-reviews_review-ldiskfs_113261.33)
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.40.34@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.40.34@tcp
LNet: There was an unexpected network error while writing to 10.240.40.34: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-91vm1.trevis.whamcloud.com to discover 10.240.40.34@tcp \(in background\)
Lustre: DEBUG MARKER: Force trevis-91vm1.trevis.whamcloud.com to discover 10.240.40.34@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.40.34@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-47vm7 to discover 10.240.43.140@tcp
Lustre: DEBUG MARKER: Force trevis-47vm7 to discover 10.240.43.140@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1136077
Lustre: DEBUG MARKER: Wait for 1136077
LNet: There was an unexpected network error while writing to 10.240.40.34: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1136077
Lustre: DEBUG MARKER: Finished wait on 1136077
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1143920:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1143920:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1143920 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? krealloc+0xa5/0xd0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? move_addr_to_user+0x4b/0xe0
? ____sys_recvmsg+0xeb/0x1b0
? __import_iovec+0x46/0x150
? import_iovec+0x17/0x20
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7f5c92f0f917

LNet: 1142520:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.24.89@tcp
LNet: Removed LNI 10.240.24.89@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.24.89@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.29.78@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.29.78@tcp
LNet: There was an unexpected network error while writing to 10.240.29.78: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-50vm2.onyx.whamcloud.com to discover 10.240.29.78@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-50vm2.onyx.whamcloud.com to discover 10.240.29.78@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.29.78@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-113vm11 to discover 10.240.24.89@tcp
Lustre: DEBUG MARKER: Force onyx-113vm11 to discover 10.240.24.89@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1143323
Lustre: DEBUG MARKER: Wait for 1143323
LNet: There was an unexpected network error while writing to 10.240.29.78: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1143323
Lustre: DEBUG MARKER: Finished wait on 1143323
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1365282:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1365282:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1365282 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.8.0+1454+0b2cbfb8 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? krealloc+0xa5/0xd0
? _copy_to_iter+0x1d4/0x630
? _copy_to_iter+0x1d4/0x630
genl_rcv_msg+0x47/0xa0
? __check_object_size.part.0+0x47/0xd0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __import_iovec+0x46/0x150
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
? idr_get_next_ul+0xb6/0xf0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? do_syscall_64+0x69/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7fb3c130f917

LNet: 1363811:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.39.124@tcp
LNet: Removed LNI 10.240.39.124@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.39.124@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.89@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.89@tcp
LNet: There was an unexpected network error while writing to 10.240.39.89: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-39vm1.trevis.whamcloud.com to discover 10.240.39.89@tcp \(in background\)
Lustre: DEBUG MARKER: Force trevis-39vm1.trevis.whamcloud.com to discover 10.240.39.89@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.89@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-37vm6 to discover 10.240.39.124@tcp
Lustre: DEBUG MARKER: Force trevis-37vm6 to discover 10.240.39.124@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1364649
Lustre: DEBUG MARKER: Wait for 1364649
LNet: There was an unexpected network error while writing to 10.240.39.89: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1364649
Lustre: DEBUG MARKER: Finished wait on 1364649
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1129184:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1129184:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1129184 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __kmem_cache_alloc_node+0x1c7/0x2d0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? _copy_to_user+0x1a/0x30
? move_addr_to_user+0x4b/0xe0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
? idr_get_next_ul+0xb6/0xf0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7fc0e770f917

LNet: 1127783:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.24.51@tcp
LNet: Removed LNI 10.240.24.51@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.24.51@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.185@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.185@tcp
LNet: There was an unexpected network error while writing to 10.240.28.185: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-48vm4.onyx.whamcloud.com to discover 10.240.28.185@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-48vm4.onyx.whamcloud.com to discover 10.240.28.185@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.185@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm2 to discover 10.240.24.51@tcp
Lustre: DEBUG MARKER: Force onyx-106vm2 to discover 10.240.24.51@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1128587
Lustre: DEBUG MARKER: Wait for 1128587
LNet: There was an unexpected network error while writing to 10.240.28.185: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1128587
Lustre: DEBUG MARKER: Finished wait on 1128587
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1140984:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1140984:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1140984 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? skb_queue_tail+0x1b/0x50
? sock_def_readable+0x10/0xc0
? __netlink_sendskb+0x67/0x90
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __import_iovec+0x46/0x150
? import_iovec+0x17/0x20
? idr_get_next_ul+0xb6/0xf0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? do_user_addr_fault+0x1d6/0x6a0
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7f8e5af0f917

LNet: 1139584:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.24.152@tcp
LNet: Removed LNI 10.240.24.152@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.24.152@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-53vm5.onyx.whamcloud.com to discover 10.240.28.193@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-53vm5.onyx.whamcloud.com to discover 10.240.28.193@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm10 to discover 10.240.24.152@tcp
Lustre: DEBUG MARKER: Force onyx-106vm10 to discover 10.240.24.152@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1140387
Lustre: DEBUG MARKER: Wait for 1140387
LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1140387
Lustre: DEBUG MARKER: Finished wait on 1140387
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1153163:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1153163:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1153163 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-503.15.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __check_object_size.part.0+0x47/0xd0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? idr_get_next_ul+0xb6/0xf0
? __wake_up+0x40/0x60
? netlink_setsockopt+0x2f7/0x430
? do_sock_setsockopt+0xb7/0x180
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? ___sys_recvmsg+0x88/0xd0
? __handle_mm_fault+0x2fb/0x690
? __sys_recvmsg+0x56/0xa0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fa52310fa17

LNet: 1151705:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.235@tcp
LNet: Removed LNI 10.240.25.235@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.235@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.24.229@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.24.229@tcp
LNet: There was an unexpected network error while writing to 10.240.24.229: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm4.onyx.whamcloud.com to discover 10.240.24.229@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm4.onyx.whamcloud.com to discover 10.240.24.229@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.24.229@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-57vm2 to discover 10.240.25.235@tcp
Lustre: DEBUG MARKER: Force onyx-57vm2 to discover 10.240.25.235@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1152542
Lustre: DEBUG MARKER: Wait for 1152542
LNet: There was an unexpected network error while writing to 10.240.24.229: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1152542
Lustre: DEBUG MARKER: Finished wait on 1152542
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1153217:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1153217:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1153217 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-503.15.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __pfx_genl_rcv_msg+0x10/0x10
? netlink_rcv_skb+0x84/0x100
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __sys_setsockopt+0x75/0xc0
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? __import_iovec+0x46/0x150
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
? ___sys_recvmsg+0x88/0xd0
? __pfx_lru_add_fn+0x10/0x10
? folio_batch_move_lru+0xd3/0x150
? __sys_recvmsg+0x56/0xa0
? __sys_recvmsg+0x56/0xa0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? handle_mm_fault+0x116/0x270
? do_user_addr_fault+0x1d6/0x6a0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7f277df0fa17

LNet: 1151761:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.238@tcp
LNet: Removed LNI 10.240.25.238@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.238@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.224@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.224@tcp
LNet: There was an unexpected network error while writing to 10.240.25.224: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm7.onyx.whamcloud.com to discover 10.240.25.224@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm7.onyx.whamcloud.com to discover 10.240.25.224@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.224@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-69vm13 to discover 10.240.25.238@tcp
Lustre: DEBUG MARKER: Force onyx-69vm13 to discover 10.240.25.238@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1152596
Lustre: DEBUG MARKER: Wait for 1152596
LNet: There was an unexpected network error while writing to 10.240.25.224: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1152596
Lustre: DEBUG MARKER: Finished wait on 1152596
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1128144:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1128144:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1128144 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.8.0+1454+0b2cbfb8 04/01/2014
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? _copy_to_iter+0x1d4/0x630
? sock_def_readable+0x10/0xc0
? _copy_to_iter+0x1d4/0x630
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
? idr_get_next_ul+0xb6/0xf0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_setsockopt+0x281/0x460
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7f32b630f917

LNet: 1126744:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.24.210@tcp
LNet: Removed LNI 10.240.24.210@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.24.210@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.195@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.195@tcp
LNet: There was an unexpected network error while writing to 10.240.28.195: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-56vm3.onyx.whamcloud.com to discover 10.240.28.195@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-56vm3.onyx.whamcloud.com to discover 10.240.28.195@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.195@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm12 to discover 10.240.24.210@tcp
Lustre: DEBUG MARKER: Force onyx-106vm12 to discover 10.240.24.210@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1127547
Lustre: DEBUG MARKER: Wait for 1127547
LNet: There was an unexpected network error while writing to 10.240.28.195: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1127547
Lustre: DEBUG MARKER: Finished wait on 1127547
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1137822:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1137822:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1137822 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? netlink_ack+0x157/0x240
? __pfx_genl_rcv_msg+0x10/0x10
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? netlink_setsockopt+0x281/0x460
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7f634c50f917

LNet: 1136422:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.43.235@tcp
LNet: Removed LNI 10.240.43.235@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.43.235@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.85@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.85@tcp
LNet: There was an unexpected network error while writing to 10.240.44.85: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-96vm4.trevis.whamcloud.com to discover 10.240.44.85@tcp \(in background\)
Lustre: DEBUG MARKER: Force trevis-96vm4.trevis.whamcloud.com to discover 10.240.44.85@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.85@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-103vm10 to discover 10.240.43.235@tcp
Lustre: DEBUG MARKER: Force trevis-103vm10 to discover 10.240.43.235@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1137225
Lustre: DEBUG MARKER: Wait for 1137225
LNet: There was an unexpected network error while writing to 10.240.44.85: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1137225
Lustre: DEBUG MARKER: Finished wait on 1137225
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1154644:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1154644:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1154644 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.42.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __kmem_cache_alloc_node+0x1c7/0x2d0
? __alloc_skb+0x8e/0x1d0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x69/0x90
entry_SYSCALL_64_after_hwframe+0x77/0xe1
RIP: 0033:0x7fb4ab50f917

LNet: 1153244:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.27.188@tcp
LNet: Removed LNI 10.240.27.188@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.27.188@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-152vm9.onyx.whamcloud.com to discover 10.240.28.193@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-152vm9.onyx.whamcloud.com to discover 10.240.28.193@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm10 to discover 10.240.27.188@tcp
Lustre: DEBUG MARKER: Force onyx-106vm10 to discover 10.240.27.188@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1154047
Lustre: DEBUG MARKER: Wait for 1154047
LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1154047
Lustre: DEBUG MARKER: Finished wait on 1154047
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1141620:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1141620:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1141620 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-503.22.1.el9_5.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit+0xdc/0x130
genl_family_rcv_msg+0x14d/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x248/0x370
netlink_sendmsg+0x206/0x440
____sys_sendmsg+0x38b/0x3b0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? netlink_recvmsg+0x212/0x290
? __check_object_size.part.0+0x35/0xd0
? _copy_to_user+0x1a/0x30
? move_addr_to_user+0x4b/0xe0
? ____sys_recvmsg+0xeb/0x1b0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5f/0xf0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_setsockopt+0x2f7/0x430
? do_sock_setsockopt+0xb7/0x180
? __sys_setsockopt+0x75/0xc0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? syscall_exit_to_user_mode+0x19/0x40
? do_syscall_64+0x6b/0xf0
? do_syscall_64+0x6b/0xf0
? do_user_addr_fault+0x1d6/0x6a0
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x78/0x80
RIP: 0033:0x7fe72f90fa17

LNet: 1140220:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.28.193@tcp
LNet: Removed LNI 10.240.28.193@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.28.193@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.130@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.130@tcp
LNet: There was an unexpected network error while writing to 10.240.23.130: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm10.onyx.whamcloud.com to discover 10.240.23.130@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-106vm10.onyx.whamcloud.com to discover 10.240.23.130@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.130@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-39vm7 to discover 10.240.28.193@tcp
Lustre: DEBUG MARKER: Force onyx-39vm7 to discover 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1141023
Lustre: DEBUG MARKER: Wait for 1141023
LNet: There was an unexpected network error while writing to 10.240.23.130: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1141023
Lustre: DEBUG MARKER: Finished wait on 1141023
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1122043:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1122043:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1122043 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __kmem_cache_alloc_node+0x1c7/0x2d0
? __alloc_skb+0x8e/0x1d0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
? idr_get_next_ul+0xb6/0xf0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7f4a1cb0f917

LNet: 1120641:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.23.207@tcp
LNet: Removed LNI 10.240.23.207@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.23.207@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.187@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.187@tcp
LNet: There was an unexpected network error while writing to 10.240.23.187: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-43vm4.onyx.whamcloud.com to discover 10.240.23.187@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-43vm4.onyx.whamcloud.com to discover 10.240.23.187@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.187@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-42vm4 to discover 10.240.23.207@tcp
Lustre: DEBUG MARKER: Force onyx-42vm4 to discover 10.240.23.207@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1121446
Lustre: DEBUG MARKER: Wait for 1121446
LNet: There was an unexpected network error while writing to 10.240.23.187: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1121446
Lustre: DEBUG MARKER: Finished wait on 1121446
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1123793:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1123793:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1123793 Comm: lnetctl Kdump: loaded Tainted: G OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? ctrl_getfamily+0x16c/0x1b0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __import_iovec+0x46/0x150
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
? ___sys_recvmsg+0x88/0xd0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_setsockopt+0x281/0x460
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7f0395d0f917

LNet: 1122391:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.27.244@tcp
LNet: Removed LNI 10.240.27.244@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.27.244@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.241@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.241@tcp
LNet: There was an unexpected network error while writing to 10.240.25.241: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-96vm5.onyx.whamcloud.com to discover 10.240.25.241@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-96vm5.onyx.whamcloud.com to discover 10.240.25.241@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.241@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm10 to discover 10.240.27.244@tcp
Lustre: DEBUG MARKER: Force onyx-70vm10 to discover 10.240.27.244@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1123196
Lustre: DEBUG MARKER: Wait for 1123196
LNet: There was an unexpected network error while writing to 10.240.25.241: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1123196
Lustre: DEBUG MARKER: Finished wait on 1123196
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1097910:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1097910:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1097910 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __check_object_size.part.0+0x47/0xd0
genl_rcv_msg+0x47/0xa0
? __pfx_simple_copy_to_iter+0x10/0x10
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7fdf4d10f917

LNet: 1096508:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.23.247@tcp
LNet: Removed LNI 10.240.23.247@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.23.247@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.207@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.207@tcp
LNet: There was an unexpected network error while writing to 10.240.23.207: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-45vm4.onyx.whamcloud.com to discover 10.240.23.207@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-45vm4.onyx.whamcloud.com to discover 10.240.23.207@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.207@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-43vm4 to discover 10.240.23.247@tcp
Lustre: DEBUG MARKER: Force onyx-43vm4 to discover 10.240.23.247@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1097313
Lustre: DEBUG MARKER: Wait for 1097313
LNet: There was an unexpected network error while writing to 10.240.23.207: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1097313
Lustre: DEBUG MARKER: Finished wait on 1097313
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1090725:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1090725:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1090725 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? netlink_rcv_skb+0x84/0x100
? _copy_to_iter+0x1d4/0x630
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7f9b0410f917

LNet: 1089323:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.27.46@tcp
LNet: Removed LNI 10.240.27.46@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.27.46@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.53@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.53@tcp
LNet: There was an unexpected network error while writing to 10.240.27.53: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-86vm7.onyx.whamcloud.com to discover 10.240.27.53@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-86vm7.onyx.whamcloud.com to discover 10.240.27.53@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.53@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-86vm14 to discover 10.240.27.46@tcp
Lustre: DEBUG MARKER: Force onyx-86vm14 to discover 10.240.27.46@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1090128
Lustre: DEBUG MARKER: Wait for 1090128
LNet: There was an unexpected network error while writing to 10.240.27.53: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1090128
Lustre: DEBUG MARKER: Finished wait on 1090128
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1104577:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1104577:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1104577 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7fb95e70f917

LNet: 1103179:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.26.99@tcp
LNet: Removed LNI 10.240.26.99@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.26.99@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.108@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.108@tcp
LNet: There was an unexpected network error while writing to 10.240.26.108: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-76vm4.onyx.whamcloud.com to discover 10.240.26.108@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-76vm4.onyx.whamcloud.com to discover 10.240.26.108@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.108@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-76vm13 to discover 10.240.26.99@tcp
Lustre: DEBUG MARKER: Force onyx-76vm13 to discover 10.240.26.99@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1103980
Lustre: DEBUG MARKER: Wait for 1103980
LNet: There was an unexpected network error while writing to 10.240.26.108: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1103980
Lustre: DEBUG MARKER: Finished wait on 1103980
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1085356:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1085356:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1085356 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? krealloc+0xa5/0xd0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? move_addr_to_user+0x4b/0xe0
? ____sys_recvmsg+0xeb/0x1b0
? __import_iovec+0x46/0x150
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_setsockopt+0x281/0x460
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7f5a6630f917

LNet: 1083955:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.28.167@tcp
LNet: Removed LNI 10.240.28.167@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.28.167@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.175@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.175@tcp
LNet: There was an unexpected network error while writing to 10.240.28.175: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-105vm4.onyx.whamcloud.com to discover 10.240.28.175@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-105vm4.onyx.whamcloud.com to discover 10.240.28.175@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.175@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-105vm12 to discover 10.240.28.167@tcp
Lustre: DEBUG MARKER: Force onyx-105vm12 to discover 10.240.28.167@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1084759
Lustre: DEBUG MARKER: Wait for 1084759
LNet: There was an unexpected network error while writing to 10.240.28.175: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1084759
Lustre: DEBUG MARKER: Finished wait on 1084759
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1102693:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1102693:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1102693 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? netlink_rcv_skb+0x84/0x100
? _copy_to_iter+0x1d4/0x630
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? _raw_spin_unlock_irqrestore+0xa/0x30
? __wake_up+0x40/0x60
? netlink_setsockopt+0x281/0x460
? __sys_setsockopt+0xdc/0x1d0
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7fa139b0f917

LNet: 1101290:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.26.139@tcp
LNet: Removed LNI 10.240.26.139@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.26.139@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.147@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.147@tcp
LNet: There was an unexpected network error while writing to 10.240.26.147: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-78vm4.onyx.whamcloud.com to discover 10.240.26.147@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-78vm4.onyx.whamcloud.com to discover 10.240.26.147@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.147@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-78vm12 to discover 10.240.26.139@tcp
Lustre: DEBUG MARKER: Force onyx-78vm12 to discover 10.240.26.139@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1102096
Lustre: DEBUG MARKER: Wait for 1102096
LNet: There was an unexpected network error while writing to 10.240.26.147: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1102096
Lustre: DEBUG MARKER: Finished wait on 1102096
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1100766:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1100766:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1100766 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.31.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? __alloc_skb+0x8e/0x1d0
? __alloc_skb+0x8e/0x1d0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x24c/0x4c0
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? __import_iovec+0x46/0x150
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
? do_syscall_64+0x69/0x90
? do_syscall_64+0x69/0x90
? exc_page_fault+0x62/0x150
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7f481c30f917

LNet: 1099364:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.39.4@tcp
LNet: Removed LNI 10.240.39.4@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.39.4@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.5@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.5@tcp
LNet: There was an unexpected network error while writing to 10.240.39.5: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-33vm1.trevis.whamcloud.com to discover 10.240.39.5@tcp \(in background\)
Lustre: DEBUG MARKER: Force trevis-33vm1.trevis.whamcloud.com to discover 10.240.39.5@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.5@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-33vm2 to discover 10.240.39.4@tcp
Lustre: DEBUG MARKER: Force trevis-33vm2 to discover 10.240.39.4@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1100169
Lustre: DEBUG MARKER: Wait for 1100169
LNet: There was an unexpected network error while writing to 10.240.39.5: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1100169
Lustre: DEBUG MARKER: Finished wait on 1100169
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

LNetError: 1062187:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1062187:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1062187 Comm: lnetctl Kdump: loaded Tainted: G W OE ------- --- 5.14.0-427.24.1.el9_4.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
<TASK>
dump_stack_lvl+0x34/0x48
lbug_with_loc.cold+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0x9c/0xd0 [lnet]
? __pfx_lnet_discovery_event_handler+0x10/0x10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x66/0x80 [lnet]
genl_family_rcv_msg_doit.isra.0+0xcb/0x120
genl_family_rcv_msg+0x14c/0x220
? __pfx_lnet_net_conf_cmd+0x10/0x10 [lnet]
? krealloc+0xa5/0xd0
genl_rcv_msg+0x47/0xa0
? __pfx_genl_rcv_msg+0x10/0x10
netlink_rcv_skb+0x57/0x100
genl_rcv+0x24/0x40
netlink_unicast+0x23e/0x360
netlink_sendmsg+0x238/0x480
____sys_sendmsg+0x31f/0x340
? import_iovec+0x17/0x20
? copy_msghdr_from_user+0x6d/0xa0
___sys_sendmsg+0x88/0xd0
? copy_msghdr_from_user+0x6d/0xa0
? __kmem_cache_alloc_node+0x1c7/0x2d0
? netlink_realloc_groups+0xbe/0x120
? idr_get_next_ul+0xb6/0xf0
__sys_sendmsg+0x59/0xa0
do_syscall_64+0x5c/0x90
? do_syscall_64+0x69/0x90
? do_syscall_64+0x69/0x90
? syscall_exit_work+0x103/0x130
? syscall_exit_to_user_mode+0x22/0x40
? do_syscall_64+0x69/0x90
entry_SYSCALL_64_after_hwframe+0x72/0xdc
RIP: 0033:0x7f4ee630f917

LNet: 1060787:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.232@tcp
LNet: Removed LNI 10.240.25.232@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.232@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.242@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.242@tcp
LNet: There was an unexpected network error while writing to 10.240.25.242: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.25.242@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.25.242@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.242@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm11 to discover 10.240.25.232@tcp
Lustre: DEBUG MARKER: Force onyx-70vm11 to discover 10.240.25.232@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1061590
Lustre: DEBUG MARKER: Wait for 1061590
LNet: There was an unexpected network error while writing to 10.240.25.242: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1061590
Lustre: DEBUG MARKER: Finished wait on 1061590
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&