| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1574144:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1574144:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 UID: 0 PID: 1574144 Comm: socknal_sd00_00 Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x42 [libcfs] ksocknal_close_conn_locked+0x682/0x690 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:d6:111d@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:7f:5e48@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:d6:111d@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:7f:5e48@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:7f:5e48: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:d6:111d@tcp LNet: 1574518:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:7f:5e48@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:7f:5e48@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:7f:5e48@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:7f:5e48@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:7f:5e48: rc = -22 LNet: Skipped 12 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1625031:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1625031:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 UID: 0 PID: 1625031 Comm: socknal_sd01_00 Kdump: loaded Tainted: G W OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x50 [libcfs] ksocknal_close_conn_locked+0x69a/0x6a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:ea:2f51@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:34:f6fe@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:ea:2f51@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:34:f6fe@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:34:f6fe: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:ea:2f51@tcp LNet: 1625404:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:34:f6fe@tcp | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1568543:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1568543:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 UID: 0 PID: 1568543 Comm: socknal_sd01_00 Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x42 [libcfs] ksocknal_close_conn_locked+0x682/0x690 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:33:e2dd@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:9a:74ca@tcp Autotest: Test running for 335 minutes (lustre-b_es-reviews_review-dne-ipv6_29386.173) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:33:e2dd@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:9a:74ca@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:9a:74ca: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:33:e2dd@tcp LNet: 1568960:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:9a:74ca@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:9a:74ca@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:9a:74ca@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:9a:74ca@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:9a:74ca: rc = -22 LNet: Skipped 11 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1624068:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1624068:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 UID: 0 PID: 1624068 Comm: socknal_sd00_00 Kdump: loaded Tainted: G W OE ------ --- 6.12.0-124.56.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x50 [libcfs] ksocknal_close_conn_locked+0x69a/0x6a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:21:a014@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:b:8118@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:21:a014@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:b:8118@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:b:8118: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:21:a014@tcp LNet: 1624442:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:b:8118@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:b:8118@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:b:8118@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:b:8118@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:b:8118: rc = -22 LNet: Skipped 11 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1630603:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1630603:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 UID: 0 PID: 1630603 Comm: socknal_sd01_00 Kdump: loaded Tainted: G W OE ------ --- 6.12.0-124.45.1.el10_1.x86_64 #1 PREEMPT(voluntary) Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x50 [libcfs] ksocknal_close_conn_locked+0x69a/0x6a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xfa/0x240 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 305 minutes (lustre-b_es-reviews_review-dne-ipv6_29294.174) Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:dc:12a2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:73:2cfc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:dc:12a2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:73:2cfc@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:73:2cfc: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:dc:12a2@tcp LNet: 1630976:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:73:2cfc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:73:2cfc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:73:2cfc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:73:2cfc@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:73:2cfc: rc = -22 LNet: Skipped 11 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1577704:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1577704:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 UID: 0 PID: 1577704 Comm: socknal_sd01_00 Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x42 [libcfs] ksocknal_close_conn_locked+0x682/0x690 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:d1:5859@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:38:43b@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:d1:5859@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:38:43b@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:38:43b: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:d1:5859@tcp LNet: 1578077:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:38:43b@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:38:43b@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:38:43b@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:38:43b@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:38:43b: rc = -22 LNet: Skipped 11 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1578628:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1578628:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 UID: 0 PID: 1578628 Comm: socknal_sd01_00 Kdump: loaded Tainted: G W OE ------- --- 6.12.0-55.41.1.el10_0.x86_64 #1 Tainted: [W]=WARN, [O]=OOT_MODULE, [E]=UNSIGNED_MODULE Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x4e/0x70 panic+0x113/0x2dd lbug_with_loc.cold+0x26/0x42 [libcfs] ksocknal_close_conn_locked+0x682/0x690 [ksocklnd] ksocknal_close_peer_conns_locked+0x4f/0x120 [ksocklnd] ksocknal_close_conn_and_siblings+0x32/0x50 [ksocklnd] ksocknal_process_receive+0x915/0xb50 [ksocklnd] ksocknal_scheduler+0x1ff/0xc80 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xcf/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x31/0x50 ? __pfx_kthread+0x10/0x10 ret_from_fork_asm+0x1a/0x30 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:0:296a@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:d5:c185@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:0:296a@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d5:c185@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:d5:c185: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:0:296a@tcp LNet: 1579001:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d5:c185@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d5:c185@tcp | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1771700:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1771700:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1771700 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.36.1.el9_7.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x50 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? srso_alias_return_thunk+0x5/0xfbef5 ? __schedule+0x239/0x540 ? lnet_attempt_msg_resend+0x137/0x3c0 [lnet] ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? prepare_to_wait_event+0x61/0x190 ksocknal_scheduler+0x1f9/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0x101/0x110 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x28/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:5d:a6be@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:a7:c5ff@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:5d:a6be@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:a7:c5ff@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:a7:c5ff: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:5d:a6be@tcp LNet: 1772073:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:a7:c5ff@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:a7:c5ff@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:a7:c5ff@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:a7:c5ff@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:a7:c5ff: rc = -22 LNet: Skipped 12 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1698677:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1698677:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1698677 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? srso_alias_return_thunk+0x5/0xfbef5 ? __schedule+0x231/0x550 ? __schedule+0x231/0x550 ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xdd/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x29/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:7e:5d41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:ba:f668@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:7e:5d41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:ba:f668@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:ba:f668: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:7e:5d41@tcp LNet: 1699051:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:ba:f668@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:ba:f668@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:ba:f668@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:ba:f668@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:ba:f668: rc = -22 LNet: Skipped 12 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1833529:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1833529:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1833529 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE ------ --- 5.14.0-611.36.1.el9_7.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x50 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? __schedule+0x239/0x540 ? lnet_attempt_msg_resend+0x137/0x3c0 [lnet] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? prepare_to_wait_event+0x61/0x190 ksocknal_scheduler+0x1f9/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0x104/0x110 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2b/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:37:28e1@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:58:b343@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:37:28e1@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:58:b343@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:58:b343: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:37:28e1@tcp LNet: 1833902:0:(api-ni.c:338:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:58:b343@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:58:b343@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:58:b343@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:58:b343@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:58:b343: rc = -22 LNet: Skipped 11 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1353670:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1353670:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1353670 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? srso_alias_return_thunk+0x5/0xfbef5 ? __schedule+0x231/0x550 ? lnet_attempt_msg_resend+0x137/0x3c0 [lnet] ? srso_alias_return_thunk+0x5/0xfbef5 ? srso_alias_return_thunk+0x5/0xfbef5 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xdd/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x29/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:67:8aed@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:23:db15@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:67:8aed@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:23:db15@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:23:db15: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:67:8aed@tcp LNet: 1353950:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:23:db15@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:23:db15@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:23:db15@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -d fd33:3981:3213:f010:0:5254:67:8aed@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:23:db15@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:23:db15: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1432769:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1432769:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1432769 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 1.16.3-2.el9_5.1 04/01/2014 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.8+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? srso_alias_return_thunk+0x5/0xfcdfd ? __schedule+0x2d9/0x870 ? kfree+0x22e/0x250 ? srso_alias_return_thunk+0x5/0xfcdfd ? srso_alias_return_thunk+0x5/0xfcdfd ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? srso_alias_return_thunk+0x5/0xfcdfd ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:99:e7b4@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:80:5e1a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:80:5e1a@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:80:5e1a: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:99:e7b4@tcp LNet: 1433048:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:80:5e1a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:80:5e1a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:80:5e1a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -d fd33:3981:3213:f010:0:5254:99:e7b4@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:80:5e1a@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:80:5e1a: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 71137:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 71137:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 71137 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? __schedule+0x231/0x550 ? _raw_spin_unlock_irqrestore+0xa/0x30 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:2:4af5@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:cd:fd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:2:4af5@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:cd:fd4@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:cd:fd4: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:2:4af5@tcp LNet: 71390:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:cd:fd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:cd:fd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:cd:fd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:cd:fd4@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:cd:fd4: rc = -22 LNet: Skipped 11 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1391444:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1391444:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1391444 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? __schedule+0x231/0x550 ? lnet_attempt_msg_resend+0x13b/0x3f0 [lnet] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:34:8ccb@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:50:5eca@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:50:5eca@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:50:5eca@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:50:5eca@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:50:5eca@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:50:5eca@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:50:5eca: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:34:8ccb@tcp LNet: 1391739:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:50:5eca@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:50:5eca@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:50:5eca@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:50:5eca@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:50:5eca@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:50:5eca@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:50:5eca@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:50:5eca@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:50:5eca@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -d fd33:3981:3213:f020:0:5254:34:8ccb@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:50:5eca@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:50:5eca: rc = -22 LNet: Skipped 6 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 70938:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 70938:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 70938 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? __schedule+0x231/0x550 ? lnet_attempt_msg_resend+0x137/0x3c0 [lnet] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 15 minutes (lustre-b_es-reviews_review-dne-ipv6_25269.106) Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:64:1d1d@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:ca:2f20@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:64:1d1d@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ca:2f20@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:ca:2f20: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:64:1d1d@tcp LNet: 71190:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ca:2f20@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ca:2f20@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:ca:2f20: rc = -22 LNet: Skipped 6 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1352515:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1352515:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1352515 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? __schedule+0x231/0x550 ? lnet_attempt_msg_resend+0x137/0x3c0 [lnet] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:a8:b2f3@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:20:8cc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:20:8cc@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:20:8cc: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:a8:b2f3@tcp LNet: 1352795:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:20:8cc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:20:8cc@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:20:8cc@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:20:8cc@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:20:8cc: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1430146:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1430146:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1430146 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.8+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0x22e/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:e5:29b2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:5e:6347@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:5e:6347@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:5e:6347@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:5e:6347@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:5e:6347@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:5e:6347@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:5e:6347: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:e5:29b2@tcp LNet: 1430426:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:5e:6347@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:5e:6347@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:5e:6347@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:5e:6347@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:5e:6347@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:5e:6347@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:5e:6347@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:5e:6347@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:5e:6347@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -d fd33:3981:3213:f020:0:5254:e5:29b2@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:5e:6347@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:5e:6347: rc = -22 LNet: Skipped 5 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1485134:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1485134:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1485134 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:b8:4912@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:47:bee6@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:b8:4912@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:47:bee6@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:47:bee6: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:b8:4912@tcp LNet: 1485428:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:47:bee6@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:47:bee6@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:47:bee6@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -d fd33:3981:3213:f020:0:5254:b8:4912@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:47:bee6@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:47:bee6: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1476767:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1476767:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1476767 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:2c:9ea7@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:4a:f8bd@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:4a:f8bd@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:4a:f8bd: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:2c:9ea7@tcp LNet: 1477063:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:4a:f8bd@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:4a:f8bd@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -d fd33:3981:3213:f020:0:5254:2c:9ea7@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:4a:f8bd@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:4a:f8bd: rc = -22 LNet: Skipped 5 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1472228:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1472228:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1472228 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:b9:dff1@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:71:2cd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:71:2cd4@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:71:2cd4: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:b9:dff1@tcp LNet: 1472522:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:71:2cd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:71:2cd4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:71:2cd4@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -d fd33:3981:3213:f010:0:5254:b9:dff1@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:71:2cd4@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:71:2cd4: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1476000:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1476000:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1476000 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:28:7cbd@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:3:8901@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:3:8901@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:3:8901@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:3:8901@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:3:8901@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:3:8901@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:3:8901: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:28:7cbd@tcp LNet: 1476294:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:3:8901@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:3:8901@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:3:8901@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:3:8901@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:3:8901@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:3:8901@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:3:8901@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:3:8901@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:3:8901@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -d fd33:3981:3213:f010:0:5254:28:7cbd@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:3:8901@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:3:8901: rc = -22 LNet: Skipped 5 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1478674:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1478674:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1478674 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:86:fca9@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:29:d732@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:86:fca9@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:29:d732@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:29:d732: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:86:fca9@tcp LNet: 1478968:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:29:d732@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:29:d732@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:29:d732@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -d fd33:3981:3213:f010:0:5254:86:fca9@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:29:d732@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:29:d732: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1474498:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1474498:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1474498 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:a8:b2f3@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:31:6857@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:31:6857@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:31:6857: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:a8:b2f3@tcp LNet: 1474792:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:31:6857@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:31:6857@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:31:6857@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -d fd33:3981:3213:f010:0:5254:a8:b2f3@tcp1 -m GET -e local_no_route -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:31:6857@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:31:6857: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1476404:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1476404:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1476404 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:d9:e97f@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:49:6cd1@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -m GET -e local_interrupt -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:49:6cd1@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:49:6cd1: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:d9:e97f@tcp LNet: 1476698:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -m GET -e local_dropped -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:49:6cd1@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:49:6cd1@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -d fd33:3981:3213:f010:0:5254:d9:e97f@tcp1 -m GET -e local_aborted -r 1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:49:6cd1@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:49:6cd1: rc = -22 LNet: Skipped 5 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1436578:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1436578:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1436578 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:27:4634@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:11:8988@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:27:4634@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:11:8988@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:11:8988: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:27:4634@tcp LNet: 1436861:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:11:8988@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:11:8988@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:11:8988@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:27:4634@tcp1 -d fd33:3981:3213:f010:0:5254:27:4634@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:11:8988@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:11:8988: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1436142:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1436142:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1436142 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:df:fd55@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:dd:4e6a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:df:fd55@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:dd:4e6a@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:dd:4e6a: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:df:fd55@tcp LNet: 1436424:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:dd:4e6a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:dd:4e6a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:dd:4e6a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -d fd33:3981:3213:f010:0:5254:df:fd55@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:dd:4e6a@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:dd:4e6a: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1436188:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1436188:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1436188 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0x22e/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:a7:a72e@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:36:cf01@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:36:cf01@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:36:cf01: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:a7:a72e@tcp LNet: 1436470:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:36:cf01@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:36:cf01@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:36:cf01@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -d fd33:3981:3213:f010:0:5254:a7:a72e@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:36:cf01@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:36:cf01: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1431985:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1431985:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1431985 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.8+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:bf:901c@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:57:cc71@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:bf:901c@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:57:cc71@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:57:cc71: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:bf:901c@tcp LNet: 1432266:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:57:cc71@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:57:cc71@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:57:cc71@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -d fd33:3981:3213:f020:0:5254:bf:901c@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:57:cc71@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:57:cc71: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1438009:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1438009:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1438009 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0x22e/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:19:2fb8@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:bb:7a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:bb:7a@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:bb:7a: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:19:2fb8@tcp LNet: 1438292:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:bb:7a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:bb:7a@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:bb:7a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -d fd33:3981:3213:f020:0:5254:19:2fb8@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:bb:7a@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:bb:7a: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1437945:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1437945:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1437945 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:5:20a@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:d1:fea4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:5:20a@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d1:fea4@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:d1:fea4: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:5:20a@tcp LNet: 1438227:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d1:fea4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d1:fea4@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:d1:fea4@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:5:20a@tcp1 -d fd33:3981:3213:f010:0:5254:5:20a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:d1:fea4@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:d1:fea4: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1441629:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1441629:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1441629 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:1e:2056@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:b1:c977@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:1e:2056@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:b1:c977@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:b1:c977: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:1e:2056@tcp LNet: 1441911:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:b1:c977@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:b1:c977@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:b1:c977@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -d fd33:3981:3213:f010:0:5254:1e:2056@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:b1:c977@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:b1:c977: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1437191:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1437191:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1437191 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0x22e/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:f0:7395@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:67:f0cf@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:f0:7395@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:67:f0cf@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:67:f0cf: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:f0:7395@tcp LNet: 1437474:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:67:f0cf@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:67:f0cf@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:67:f0cf@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -d fd33:3981:3213:f020:0:5254:f0:7395@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:67:f0cf@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:67:f0cf: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1434880:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1434880:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1434880 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:4c:3906@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:7f:2006@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:4c:3906@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:7f:2006@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:7f:2006: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:4c:3906@tcp LNet: 1435162:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:7f:2006@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:7f:2006@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:7f:2006@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -d fd33:3981:3213:f010:0:5254:4c:3906@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:7f:2006@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:7f:2006: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1436466:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1436466:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1436466 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.8+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0x22e/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f010:0:5254:f5:5e6b@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f010:0:5254:50:e412@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:50:e412@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:50:e412@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:50:e412@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:50:e412@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:50:e412@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:50:e412: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f010:0:5254:f5:5e6b@tcp LNet: 1436746:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:50:e412@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:50:e412@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:50:e412@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:50:e412@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:50:e412@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:50:e412@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:50:e412@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:50:e412@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:50:e412@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -d fd33:3981:3213:f010:0:5254:f5:5e6b@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f010:0:5254:50:e412@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f010:0:5254:50:e412: rc = -22 LNet: Skipped 5 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1440493:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1440493:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1440493 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:b5:613a@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:8:b611@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:b5:613a@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8:b611@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:8:b611: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:b5:613a@tcp LNet: 1440776:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8:b611@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8:b611@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:8:b611@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -d fd33:3981:3213:f020:0:5254:b5:613a@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8:b611@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:8:b611: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 75842:0:(socklnd.c:1382:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 75842:0:(socklnd.c:1382:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 75842 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: <TASK> dump_stack_lvl+0x34/0x48 panic+0x107/0x2bb lbug_with_loc.cold+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x5e2/0x610 [ksocklnd] ksocknal_close_peer_conns_locked+0x7a/0x110 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x959/0xb70 [ksocklnd] ? __schedule+0x231/0x550 ? lnet_attempt_msg_resend+0x137/0x3c0 [lnet] ? _raw_spin_unlock_irqrestore+0xa/0x30 ? prepare_to_wait_event+0x5d/0x180 ksocknal_scheduler+0x1fa/0xc20 [ksocklnd] ? __pfx_autoremove_wake_function+0x10/0x10 ? __pfx_ksocknal_scheduler+0x10/0x10 [ksocklnd] kthread+0xe0/0x100 ? __pfx_kthread+0x10/0x10 ret_from_fork+0x2c/0x50 </TASK> | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:f3:2a44@tcp [8/256/0/180] LNet: Accept all, port 7988 Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:1:5dc5@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:f3:2a44@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:1:5dc5@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:1:5dc5: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:f3:2a44@tcp LNet: 76095:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:1:5dc5@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:1:5dc5@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:1:5dc5@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:1:5dc5@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:1:5dc5: rc = -22 LNet: Skipped 12 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1432193:0:(socklnd.c:1355:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1432193:0:(socklnd.c:1355:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 1 PID: 1432193 Comm: socknal_sd01_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:c8:3c84@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:9:fb11@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:9:fb11@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:9:fb11: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:c8:3c84@tcp LNet: 1432475:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:9:fb11@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:9:fb11@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:9:fb11@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -d fd33:3981:3213:f020:0:5254:c8:3c84@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:9:fb11@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:9:fb11: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1434444:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1434444:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1434444 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:20:96a9@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:98:8d76@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:20:96a9@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:98:8d76@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:98:8d76: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:20:96a9@tcp LNet: 1434724:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:98:8d76@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:98:8d76@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:98:8d76@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -d fd33:3981:3213:f020:0:5254:20:96a9@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:98:8d76@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:98:8d76: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1436069:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1436069:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1436069 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:82:e70c@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:8a:97f1@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:82:e70c@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8a:97f1@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:8a:97f1: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:82:e70c@tcp LNet: 1436349:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8a:97f1@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8a:97f1@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:8a:97f1@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -d fd33:3981:3213:f020:0:5254:82:e70c@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:8a:97f1@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:8a:97f1: rc = -22 LNet: Skipped 9 previous similar messages | Link to test |
| sanity-lnet test 205: Check health and resends for multi-rail local failures | LNetError: 1433916:0:(socklnd.c:1368:ksocknal_close_conn_locked()) ASSERTION( conn->ksnc_proto == &ksocknal_protocol_v3x ) failed: LNetError: 1433916:0:(socklnd.c:1368:ksocknal_close_conn_locked()) LBUG Kernel panic - not syncing: LBUG in interrupt. CPU: 0 PID: 1433916 Comm: socknal_sd00_00 Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.58.1.el8_10.x86_64 #1 Hardware name: Red Hat KVM, BIOS 0.5.1 01/01/2011 Call Trace: dump_stack+0x41/0x60 panic+0xe7/0x2ac lbug_with_loc.cold.6+0x26/0x43 [libcfs] ksocknal_close_conn_locked+0x581/0x5a0 [ksocklnd] ksocknal_close_peer_conns_locked+0x77/0x100 [ksocklnd] ksocknal_close_conn_and_siblings+0x30/0x50 [ksocklnd] ksocknal_process_receive+0x2cd/0xf90 [ksocklnd] ? __schedule+0x2d9/0x870 ? kfree+0xd3/0x250 ? prepare_to_wait_event+0xa0/0x180 ksocknal_scheduler+0x188/0x1820 [ksocklnd] ? finish_wait+0x80/0x80 ? ksocknal_recv+0x2a0/0x2a0 [ksocklnd] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: lsmod | grep lnet > /dev/null && Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 2 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -a -l LNet: Added LNI fd33:3981:3213:f020:0:5254:b9:9bf7@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover fd33:3981:3213:f020:0:5254:ee:9151@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_interrupt Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ee:9151@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:ee:9151: rc = -22 LNet: 1 local NIs in recovery (showing 1): fd33:3981:3213:f020:0:5254:b9:9bf7@tcp LNet: 1434196:0:(api-ni.c:337:recovery_interval_set()) 'lnet_recovery_interval' has been deprecated Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_dropped Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ee:9151@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_aborted Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ee:9151@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --health 1000 --all Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp --lnd-timeout 3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:ee:9151@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl net_drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl fault drop add -s fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -d fd33:3981:3213:f020:0:5254:b9:9bf7@tcp1 -m GET -r 1 -e local_no_route Lustre: DEBUG MARKER: /usr/sbin/lnetctl ping fd33:3981:3213:f020:0:5254:ee:9151@tcp LNet: There was an unexpected network error while writing to fd33:3981:3213:f020:0:5254:ee:9151: rc = -22 LNet: Skipped 8 previous similar messages | Link to test |