Edit new crash report

Editing crashreport #72296

Reason	Crashing Function	Where to cut Backtrace	Reports Count
ASSERTION( md->md_handler != handler ) failed	lnet_assert_handler_unused	lnet_assert_handler_unused LNetNIFini lnet_unconfigure genl_family_rcv_msg_doit genl_family_rcv_msg genl_rcv_msg netlink_rcv_skb genl_rcv netlink_unicast netlink_sendmsg __sock_sendmsg ____sys_sendmsg ___sys_sendmsg __sys_sendmsg do_syscall_64 entry_SYSCALL_64_after_hwframe	8

Reason

Crashing Function

Where to cut Backtrace

Reports Count

ASSERTION( md->md_handler != handler ) failed

lnet_assert_handler_unused

lnet_assert_handler_unused
LNetNIFini
lnet_unconfigure
genl_family_rcv_msg_doit
genl_family_rcv_msg
genl_rcv_msg
netlink_rcv_skb
genl_rcv
netlink_unicast
netlink_sendmsg
__sock_sendmsg
____sys_sendmsg
___sys_sendmsg
__sys_sendmsg
do_syscall_64
entry_SYSCALL_64_after_hwframe

Added fields:

Match messages in logs
(every line would be required to be present in log output
Copy from "Messages before crash" column below):

Match messages in full crash
(every line would be required to be present in crash log output
Copy from "Full Crash" column below):

Limit to a test:
(Copy from below "Failing text"):

Delete these reports as invalid (real bug in review or some such)

Bug or comment:

Extra info:

Failures list (last 100):

Failing Test	Full Crash	Messages before crash	Comment
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 1227751:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 1227751:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 0 PID: 1227751 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.53.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x43 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7f4c4b49dc08	LNet: 1226142:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.28.171@tcp LNet: Removed LNI 10.240.28.171@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.28.171@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.114@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s @tcp -d @tcp -r 1 -e local_error Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.114@tcp LNet: There was an unexpected network error while writing to 10.240.28.114: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_delay add -s @tcp -d 10.240.28.114@tcp -r 1 -m GET -l 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s @tcp -d 10.240.28.114@tcp -r 1 -m GET -e local_error Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.28.114@tcp -r 1 -m PUT -l 6 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-105vm8.onyx.whamcloud.com to discover 10.240.28.114@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-105vm8.onyx.whamcloud.com to discover 10.240.28.114@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.114@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-102vm11 to discover 10.240.28.171@tcp Lustre: DEBUG MARKER: Force onyx-102vm11 to discover 10.240.28.171@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1227050 Lustre: DEBUG MARKER: Wait for 1227050 LNet: There was an unexpected network error while writing to 10.240.28.114: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1227050 Lustre: DEBUG MARKER: Finished wait on 1227050 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 1262452:0:(lib-md.c:302:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 1262452:0:(lib-md.c:302:lnet_assert_handler_unused()) LBUG CPU: 1 PID: 1262452 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x43 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7fa0cce9cc08	LNet: 1260849:0:(lib-ptl.c:969:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.24.92@tcp LNet: Removed LNI 10.240.24.92@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.24.92@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-50vm5.onyx.whamcloud.com to discover 10.240.28.193@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-50vm5.onyx.whamcloud.com to discover 10.240.28.193@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm10 to discover 10.240.24.92@tcp Lustre: DEBUG MARKER: Force onyx-106vm10 to discover 10.240.24.92@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1261753 Lustre: DEBUG MARKER: Wait for 1261753 LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1261753 Lustre: DEBUG MARKER: Finished wait on 1261753 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 1307249:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 1307249:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 1 PID: 1307249 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x43 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7f1d78d16c08	LNet: 1305640:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.42.116@tcp LNet: Removed LNI 10.240.42.116@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.42.116@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.233@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s @tcp -d @tcp -r 1 -e local_error Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.233@tcp LNet: There was an unexpected network error while writing to 10.240.39.233: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s @tcp -d 10.240.39.233@tcp -r 1 -m GET -l 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s @tcp -d 10.240.39.233@tcp -r 1 -m GET -e local_error Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.39.233@tcp -r 1 -m PUT -l 6 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-77vm1.trevis.whamcloud.com to discover 10.240.39.233@tcp \(in background\) Lustre: DEBUG MARKER: Force trevis-77vm1.trevis.whamcloud.com to discover 10.240.39.233@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.233@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-44vm10 to discover 10.240.42.116@tcp Lustre: DEBUG MARKER: Force trevis-44vm10 to discover 10.240.42.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1306548 Lustre: DEBUG MARKER: Wait for 1306548 LNet: There was an unexpected network error while writing to 10.240.39.233: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1306548 Lustre: DEBUG MARKER: Finished wait on 1306548 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 88347:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 88347:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 1 PID: 88347 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.46.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x43 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7f0267a96c08	LNet: 86745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.25.232@tcp LNet: Removed LNI 10.240.25.232@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.25.232@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.129@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.129@tcp LNet: There was an unexpected network error while writing to 10.240.23.129: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.23.129@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.23.129@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.129@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-39vm6 to discover 10.240.25.232@tcp Lustre: DEBUG MARKER: Force onyx-39vm6 to discover 10.240.25.232@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 87648 Lustre: DEBUG MARKER: Wait for 87648 LNet: There was an unexpected network error while writing to 10.240.23.129: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 87648 Lustre: DEBUG MARKER: Finished wait on 87648 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 226291:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 226291:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 0 PID: 226291 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.44.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x43 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb10/0xb10 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7f9301fadc08	LNet: 224761:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.25.238@tcp LNet: Removed LNI 10.240.25.238@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.25.238@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.61@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.61@tcp LNet: There was an unexpected network error while writing to 10.240.25.61: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm7.onyx.whamcloud.com to discover 10.240.25.61@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-70vm7.onyx.whamcloud.com to discover 10.240.25.61@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.61@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-135vm10 to discover 10.240.25.238@tcp Lustre: DEBUG MARKER: Force onyx-135vm10 to discover 10.240.25.238@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 225628 Lustre: DEBUG MARKER: Wait for 225628 LNet: There was an unexpected network error while writing to 10.240.25.61: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 225628 Lustre: DEBUG MARKER: Finished wait on 225628 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 81046:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 81046:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 1 PID: 81046 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.16.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb00/0xb00 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x58 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb00/0xb00 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7f5f51811c08	LNet: 79511:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.43.235@tcp LNet: Removed LNI 10.240.43.235@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.43.235@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.48@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.48@tcp LNet: There was an unexpected network error while writing to 10.240.44.48: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-96vm4.trevis.whamcloud.com to discover 10.240.44.48@tcp \(in background\) Lustre: DEBUG MARKER: Force trevis-96vm4.trevis.whamcloud.com to discover 10.240.44.48@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.48@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-102vm5 to discover 10.240.43.235@tcp Lustre: DEBUG MARKER: Force trevis-102vm5 to discover 10.240.43.235@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 80383 Lustre: DEBUG MARKER: Wait for 80383 LNet: There was an unexpected network error while writing to 10.240.44.48: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 80383 Lustre: DEBUG MARKER: Finished wait on 80383 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 1199884:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 1199884:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 0 PID: 1199884 Comm: lnetctl Kdump: loaded Tainted: G W OE -------- - - 4.18.0-553.16.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb00/0xb00 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x58 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb00/0xb00 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7fbe85d7bc08	LNet: 1198351:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.27.49@tcp LNet: Removed LNI 10.240.27.49@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.27.49@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.223@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.223@tcp LNet: There was an unexpected network error while writing to 10.240.26.223: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-86vm10.onyx.whamcloud.com to discover 10.240.26.223@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-86vm10.onyx.whamcloud.com to discover 10.240.26.223@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.223@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-82vm8 to discover 10.240.27.49@tcp Lustre: DEBUG MARKER: Force onyx-82vm8 to discover 10.240.27.49@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1199221 Lustre: DEBUG MARKER: Wait for 1199221 LNet: There was an unexpected network error while writing to 10.240.26.223: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1199221 Lustre: DEBUG MARKER: Finished wait on 1199221 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test
sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)	LNetError: 1168713:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed: LNetError: 1168713:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG CPU: 1 PID: 1168713 Comm: lnetctl Kdump: loaded Tainted: G W OE -------- - - 4.18.0-553.16.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: ? lnet_discovery_event_reply+0xb00/0xb00 [lnet] dump_stack+0x41/0x60 lbug_with_loc.cold.8+0x5/0x58 [libcfs] lnet_assert_handler_unused+0xa0/0xd0 [lnet] ? lnet_discovery_event_reply+0xb00/0xb00 [lnet] LNetNIFini+0x9f/0x150 [lnet] lnet_unconfigure+0x61/0x80 [lnet] genl_family_rcv_msg_doit.isra.17+0x113/0x150 genl_family_rcv_msg+0xb7/0x170 ? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet] genl_rcv_msg+0x47/0xa0 ? genl_family_rcv_msg+0x170/0x170 netlink_rcv_skb+0x54/0x110 genl_rcv+0x24/0x40 netlink_unicast+0x19a/0x230 netlink_sendmsg+0x204/0x3d0 __sock_sendmsg+0x50/0x60 ____sys_sendmsg+0x22a/0x250 ? copy_msghdr_from_user+0x5c/0x90 ? ____sys_recvmsg+0xb0/0x150 ___sys_sendmsg+0x7c/0xc0 ? copy_msghdr_from_user+0x5c/0x90 ? ___sys_recvmsg+0x89/0xc0 ? __wake_up_common_lock+0x89/0xc0 __sys_sendmsg+0x57/0xa0 do_syscall_64+0x5b/0x1a0 entry_SYSCALL_64_after_hwframe+0x66/0xcb RIP: 0033:0x7f375acbec08	LNet: 1167180:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNet: Removed LNI 10.240.26.142@tcp LNet: Removed LNI 10.240.26.142@tcp1 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a LNet: Added LNI 10.240.26.142@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.211@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.211@tcp LNet: There was an unexpected network error while writing to 10.240.28.211: rc = -22 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-78vm7.onyx.whamcloud.com to discover 10.240.28.211@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-78vm7.onyx.whamcloud.com to discover 10.240.28.211@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.211@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-107vm8 to discover 10.240.26.142@tcp Lustre: DEBUG MARKER: Force onyx-107vm8 to discover 10.240.26.142@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1168050 LNet: There was an unexpected network error while writing to 10.240.28.211: rc = -22 Lustre: DEBUG MARKER: Wait for 1168050 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1168050 Lustre: DEBUG MARKER: Finished wait on 1168050 Lustre: DEBUG MARKER: lsmod \| grep lnet > /dev/null &&	Link to test

Failing Test

Full Crash

Messages before crash

Comment

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 1227751:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1227751:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1227751 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.53.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7f4c4b49dc08

LNet: 1226142:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.28.171@tcp
LNet: Removed LNI 10.240.28.171@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.28.171@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.114@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.114@tcp
LNet: There was an unexpected network error while writing to 10.240.28.114: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_delay add -s *@tcp -d 10.240.28.114@tcp -r 1 -m GET -l 3
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s *@tcp -d 10.240.28.114@tcp -r 1 -m GET -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.28.114@tcp -r 1 -m PUT -l 6
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-105vm8.onyx.whamcloud.com to discover 10.240.28.114@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-105vm8.onyx.whamcloud.com to discover 10.240.28.114@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.114@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-102vm11 to discover 10.240.28.171@tcp
Lustre: DEBUG MARKER: Force onyx-102vm11 to discover 10.240.28.171@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1227050
Lustre: DEBUG MARKER: Wait for 1227050
LNet: There was an unexpected network error while writing to 10.240.28.114: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1227050
Lustre: DEBUG MARKER: Finished wait on 1227050
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_delay del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 1262452:0:(lib-md.c:302:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1262452:0:(lib-md.c:302:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1262452 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7fa0cce9cc08

LNet: 1260849:0:(lib-ptl.c:969:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.24.92@tcp
LNet: Removed LNI 10.240.24.92@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.24.92@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-50vm5.onyx.whamcloud.com to discover 10.240.28.193@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-50vm5.onyx.whamcloud.com to discover 10.240.28.193@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.193@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-106vm10 to discover 10.240.24.92@tcp
Lustre: DEBUG MARKER: Force onyx-106vm10 to discover 10.240.24.92@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1261753
Lustre: DEBUG MARKER: Wait for 1261753
LNet: There was an unexpected network error while writing to 10.240.28.193: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1261753
Lustre: DEBUG MARKER: Finished wait on 1261753
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 1307249:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1307249:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1307249 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.50.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7f1d78d16c08

LNet: 1305640:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.42.116@tcp
LNet: Removed LNI 10.240.42.116@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.42.116@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.233@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s *@tcp -d *@tcp -r 1 -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.233@tcp
LNet: There was an unexpected network error while writing to 10.240.39.233: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.39.233@tcp -r 1 -m GET -l 3
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s *@tcp -d 10.240.39.233@tcp -r 1 -m GET -e local_error
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay add -s *@tcp -d 10.240.39.233@tcp -r 1 -m PUT -l 6
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-77vm1.trevis.whamcloud.com to discover 10.240.39.233@tcp \(in background\)
Lustre: DEBUG MARKER: Force trevis-77vm1.trevis.whamcloud.com to discover 10.240.39.233@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.39.233@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-44vm10 to discover 10.240.42.116@tcp
Lustre: DEBUG MARKER: Force trevis-44vm10 to discover 10.240.42.116@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1306548
Lustre: DEBUG MARKER: Wait for 1306548
LNet: There was an unexpected network error while writing to 10.240.39.233: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1306548
Lustre: DEBUG MARKER: Finished wait on 1306548
Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault delay del -a
Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 88347:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 88347:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 88347 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.46.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7f0267a96c08

LNet: 86745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.232@tcp
LNet: Removed LNI 10.240.25.232@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.232@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.129@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.129@tcp
LNet: There was an unexpected network error while writing to 10.240.23.129: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.23.129@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm1.onyx.whamcloud.com to discover 10.240.23.129@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.23.129@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-39vm6 to discover 10.240.25.232@tcp
Lustre: DEBUG MARKER: Force onyx-39vm6 to discover 10.240.25.232@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 87648
Lustre: DEBUG MARKER: Wait for 87648
LNet: There was an unexpected network error while writing to 10.240.23.129: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 87648
Lustre: DEBUG MARKER: Finished wait on 87648
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 226291:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 226291:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 226291 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.44.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x43 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb10/0xb10 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7f9301fadc08

LNet: 224761:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.25.238@tcp
LNet: Removed LNI 10.240.25.238@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.25.238@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.61@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.61@tcp
LNet: There was an unexpected network error while writing to 10.240.25.61: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-70vm7.onyx.whamcloud.com to discover 10.240.25.61@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-70vm7.onyx.whamcloud.com to discover 10.240.25.61@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.61@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-135vm10 to discover 10.240.25.238@tcp
Lustre: DEBUG MARKER: Force onyx-135vm10 to discover 10.240.25.238@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 225628
Lustre: DEBUG MARKER: Wait for 225628
LNet: There was an unexpected network error while writing to 10.240.25.61: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 225628
Lustre: DEBUG MARKER: Finished wait on 225628
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 81046:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 81046:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 81046 Comm: lnetctl Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.16.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb00/0xb00 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb00/0xb00 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7f5f51811c08

LNet: 79511:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.43.235@tcp
LNet: Removed LNI 10.240.43.235@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.43.235@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.48@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.48@tcp
LNet: There was an unexpected network error while writing to 10.240.44.48: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-96vm4.trevis.whamcloud.com to discover 10.240.44.48@tcp \(in background\)
Lustre: DEBUG MARKER: Force trevis-96vm4.trevis.whamcloud.com to discover 10.240.44.48@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.44.48@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force trevis-102vm5 to discover 10.240.43.235@tcp
Lustre: DEBUG MARKER: Force trevis-102vm5 to discover 10.240.43.235@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 80383
Lustre: DEBUG MARKER: Wait for 80383
LNet: There was an unexpected network error while writing to 10.240.44.48: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 80383
Lustre: DEBUG MARKER: Finished wait on 80383
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 1199884:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1199884:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 0 PID: 1199884 Comm: lnetctl Kdump: loaded Tainted: G W OE -------- - - 4.18.0-553.16.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb00/0xb00 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb00/0xb00 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7fbe85d7bc08

LNet: 1198351:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.27.49@tcp
LNet: Removed LNI 10.240.27.49@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.27.49@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.223@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.223@tcp
LNet: There was an unexpected network error while writing to 10.240.26.223: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-86vm10.onyx.whamcloud.com to discover 10.240.26.223@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-86vm10.onyx.whamcloud.com to discover 10.240.26.223@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.223@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-82vm8 to discover 10.240.27.49@tcp
Lustre: DEBUG MARKER: Force onyx-82vm8 to discover 10.240.27.49@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1199221
Lustre: DEBUG MARKER: Wait for 1199221
LNet: There was an unexpected network error while writing to 10.240.26.223: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1199221
Lustre: DEBUG MARKER: Finished wait on 1199221
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

sanity-lnet test 212: Check discovery refcount loss bug (LU-14627)

LNetError: 1168713:0:(lib-md.c:259:lnet_assert_handler_unused()) ASSERTION( md->md_handler != handler ) failed:
LNetError: 1168713:0:(lib-md.c:259:lnet_assert_handler_unused()) LBUG
CPU: 1 PID: 1168713 Comm: lnetctl Kdump: loaded Tainted: G W OE -------- - - 4.18.0-553.16.1.el8_10.x86_64 #1
Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011
Call Trace:
? lnet_discovery_event_reply+0xb00/0xb00 [lnet]
dump_stack+0x41/0x60
lbug_with_loc.cold.8+0x5/0x58 [libcfs]
lnet_assert_handler_unused+0xa0/0xd0 [lnet]
? lnet_discovery_event_reply+0xb00/0xb00 [lnet]
LNetNIFini+0x9f/0x150 [lnet]
lnet_unconfigure+0x61/0x80 [lnet]
genl_family_rcv_msg_doit.isra.17+0x113/0x150
genl_family_rcv_msg+0xb7/0x170
? lnet_mark_ping_buffer_for_update+0x30/0x30 [lnet]
genl_rcv_msg+0x47/0xa0
? genl_family_rcv_msg+0x170/0x170
netlink_rcv_skb+0x54/0x110
genl_rcv+0x24/0x40
netlink_unicast+0x19a/0x230
netlink_sendmsg+0x204/0x3d0
__sock_sendmsg+0x50/0x60
____sys_sendmsg+0x22a/0x250
? copy_msghdr_from_user+0x5c/0x90
? ____sys_recvmsg+0xb0/0x150
___sys_sendmsg+0x7c/0xc0
? copy_msghdr_from_user+0x5c/0x90
? ___sys_recvmsg+0x89/0xc0
? __wake_up_common_lock+0x89/0xc0
__sys_sendmsg+0x57/0xa0
do_syscall_64+0x5b/0x1a0
entry_SYSCALL_64_after_hwframe+0x66/0xcb
RIP: 0033:0x7f375acbec08

LNet: 1167180:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit
LNet: Removed LNI 10.240.26.142@tcp
LNet: Removed LNI 10.240.26.142@tcp1
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&
Key type .llcrypt unregistered
Key type ._llcrypt unregistered
Key type ._llcrypt registered
Key type .llcrypt registered
libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2
alg: No test for adler32 (adler32-zlib)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a
LNet: Added LNI 10.240.26.142@tcp [8/256/0/180]
LNet: Accept all, port 7988
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery
Lustre: DEBUG MARKER: Initial discovery
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.211@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.211@tcp
LNet: There was an unexpected network error while writing to 10.240.28.211: rc = -22
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-78vm7.onyx.whamcloud.com to discover 10.240.28.211@tcp \(in background\)
Lustre: DEBUG MARKER: Force onyx-78vm7.onyx.whamcloud.com to discover 10.240.28.211@tcp (in background)
Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.28.211@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-107vm8 to discover 10.240.26.142@tcp
Lustre: DEBUG MARKER: Force onyx-107vm8 to discover 10.240.26.142@tcp
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1168050
LNet: There was an unexpected network error while writing to 10.240.28.211: rc = -22
Lustre: DEBUG MARKER: Wait for 1168050
Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1168050
Lustre: DEBUG MARKER: Finished wait on 1168050
Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null &&

Link to test

Return to new crashes list