| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Link and date |
|---|---|---|---|
| sanity-lnet test 100: Add route with single gw | LNetError: 1364098:0:(lib-move.c:1047:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 1364098:0:(lib-move.c:1047:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 1364098 Comm: monitor_thread Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.155.1.el8_10_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] lnet_return_rx_credits_locked+0x3ac/0x550 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff lnet_msg_decommit+0x2f3/0x7d0 [lnet] lnet_finalize+0x3c6/0xb80 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? lnet_ni_recv+0xef/0x330 [lnet] lnet_drop_routed_msgs_locked+0xac/0xe0 [lnet] ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_rtrpool_free_bufs+0xaa/0x160 [lnet] ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_rtrpools_free+0x52/0x90 [lnet] lnet_check_routers+0x5f6/0xad0 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? wait_for_completion_interruptible_timeout+0xa6/0x130 ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_monitor_thread+0x4f7/0x670 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? update_load_avg+0x7e/0x710 ? srso_alias_return_thunk+0x5/0xfcdff ? update_load_avg+0x7e/0x710 ? srso_alias_return_thunk+0x5/0xfcdff ? set_next_entity+0xb5/0x1e0 ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff ? finish_task_switch+0x86/0x2f0 ? srso_alias_return_thunk+0x5/0xfcdff ? __schedule+0x2d9/0x870 ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1205815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1205815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1207784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1207784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1210925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1210925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1213472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1213472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1216413:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1216413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNetError: 1216413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1216413:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1219204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1219204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1222145:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1222145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNetError: 1222145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1222145:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1224446:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1224446:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1227387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1227387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNetError: 1227387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1227387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1230668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1230668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1233610:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1233610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNetError: 1233610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1233610:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 855 minutes (lustre-master-next_full-dne-part-1_1012.265) | 2026-09-05 13:36:06 Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1240632:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1240632:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.2@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1244730:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1244730:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1246703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1246703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.2@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1250802:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1250802:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1252966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1252966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.2@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1256819:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1256819:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-180vm37.onyx.whamcloud.com to discover 10.240.22.2@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-180vm37.onyx.whamcloud.com to discover 10.240.22.2@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-159vm3 to discover 10.240.27.174@tcp Lustre: DEBUG MARKER: Force onyx-159vm3 to discover 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1672298 Lustre: DEBUG MARKER: Wait for 1672298 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1672298 Lustre: DEBUG MARKER: Finished wait on 1672298 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1260049:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1260049:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 860 minutes (lustre-master-next_full-dne-part-1_1012.265) | 2026-09-05 13:41:06 Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1264107:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1264107:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1267037:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1267037:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1267037:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1267336:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1267336:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1269638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1269638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1272568:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1272568:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1272568:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1272568:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1272675:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1272675:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1274976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1274976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1277907:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1277907:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1277907:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1280702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1280702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1283003:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1283003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1285933:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1285933:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1285933:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1285933:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1288728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1288728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1291029:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1291029:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 865 minutes (lustre-master-next_full-dne-part-1_1012.265) | 2026-09-05 13:46:07 Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1294010:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1294010:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1294010:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1296613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1296613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1298914:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1298914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1301844:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1301844:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1301844:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1301844:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1304639:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1304639:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1307135:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1307135:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1310353:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1310353:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1310353:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1310460:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1310460:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1312953:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1312953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1316075:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1316075:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1316075:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1316075:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1316278:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1316278:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1318771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1318771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1321990:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1321990:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1321990:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Autotest: Test running for 870 minutes (lustre-master-next_full-dne-part-1_1012.265) | 2026-09-05 13:51:08 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1322341:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1322341:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.2@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1327131:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1327131:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1329966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1329966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1332896:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1332896:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1332896:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.22.2@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp | grep -E -A 1 '\s+nid: 10.240.27.174@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp3 | grep -E -A 1 '\s+nid: 10.240.27.174@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp | grep -E -A 1 '\s+nid: 10.240.27.174@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp3 | grep -E -A 1 '\s+nid: 10.240.27.174@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp | grep -E -A 1 '\s+nid: 10.240.27.174@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp3 | grep -E -A 1 '\s+nid: 10.240.27.174@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1334368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1334368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 LNetError: 1334368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1334368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.22.2@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1336669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1336669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.22.2@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.22.2@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1341249:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1341249:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1341249:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: 1 peer NIs in recovery (showing 1): 10.240.27.175@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 3 previous similar messages LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 188 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1342128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1342128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 LNetError: 1342128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1342128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.22.2@tcp2 LNet: Removed LNI 10.240.22.2@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1344429:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1344429:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.22.2@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.22.2@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1349008:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1349008:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1349008:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.174@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.175@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.27.175@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 48 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 152 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1349983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1349983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNetError: 1349983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1349983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.22.2@tcp1 LNet: Removed LNI 10.240.22.2@tcp2 LNet: Removed LNI 10.240.22.2@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 875 minutes (lustre-master-next_full-dne-part-1_1012.265) | 2026-09-05 13:56:08 Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 1352891:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1352891:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1355619:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1355619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1358549:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1358549:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1358549:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.27.174@tcp -d 10.240.27.175@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.27.175@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1360385:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1360385:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp LNet: Removed LNI 10.240.22.2@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1362878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1362878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.2@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.2@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-159vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-159vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.2@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm38.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm38.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1366096:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1366096:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1366096:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm9.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm9.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm37.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm37.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-159vm3 Lustre: DEBUG MARKER: Disable routing on onyx-159vm3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test (2026-09-05 13:22) |
| sanity-lnet test 100: Add route with single gw | LNetError: 1079840:0:(lib-move.c:1047:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 1079840:0:(lib-move.c:1047:lnet_return_rx_credits_locked()) LBUG CPU: 0 PID: 1079840 Comm: monitor_thread Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.155.1.el8_10_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] lnet_return_rx_credits_locked+0x3ac/0x550 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff lnet_msg_decommit+0x2e9/0x7c0 [lnet] lnet_finalize+0x3c6/0xb40 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? lnet_ni_recv+0xef/0x330 [lnet] lnet_drop_routed_msgs_locked+0xac/0xe0 [lnet] ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_rtrpool_free_bufs+0xaa/0x160 [lnet] ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_rtrpools_free+0x52/0x90 [lnet] lnet_check_routers+0x5f6/0xad0 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? wait_for_completion_interruptible_timeout+0xa6/0x130 ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_monitor_thread+0x4f7/0x670 [lnet] ? srso_alias_return_thunk+0x5/0xfcdff ? update_load_avg+0x7e/0x710 ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff ? newidle_balance+0x2b6/0x3b0 ? srso_alias_return_thunk+0x5/0xfcdff ? srso_alias_return_thunk+0x5/0xfcdff ? finish_task_switch+0x86/0x2f0 ? srso_alias_return_thunk+0x5/0xfcdff ? __schedule+0x2d9/0x870 ? srso_alias_return_thunk+0x5/0xfcdff ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x1f/0x40 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 852449:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 852449:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 855547:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 855547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 860380:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 860380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNetError: 860380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 860380:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 864054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 864054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Autotest: Test running for 405 minutes (lustre-reviews_review-ldiskfs_130515.24) | 2026-08-29 01:48:37 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 868739:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 868739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNetError: 868739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 868739:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 872658:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 872658:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 877291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 877291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 880720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 880720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 885355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 885355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 889764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 889764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 894397:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 894397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 901917:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 901917:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.28.182@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 907384:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 907384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 909894:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 909894:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.28.182@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 915358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 915358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 918060:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 918060:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Autotest: Test running for 410 minutes (lustre-reviews_review-ldiskfs_130515.24) | 2026-08-29 01:53:38 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.28.182@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 923331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 923331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-157vm102.onyx.whamcloud.com to discover 10.240.28.182@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-157vm102.onyx.whamcloud.com to discover 10.240.28.182@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-156vm227 to discover 10.240.27.230@tcp Lustre: DEBUG MARKER: Force onyx-156vm227 to discover 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1431749 Lustre: DEBUG MARKER: Wait for 1431749 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1431749 Lustre: DEBUG MARKER: Finished wait on 1431749 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 927099:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 927099:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 932220:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 932220:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 936842:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 936842:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 936842:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 937717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 937717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNetError: 937717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 937717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 941146:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 941146:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 945768:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 945768:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 945768:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 945768:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 945875:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 945875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 949304:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 949304:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 953926:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 953926:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 953926:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp LNet: 1 peer NIs in recovery (showing 1): 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 956529:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 956529:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 415 minutes (lustre-reviews_review-ldiskfs_130515.24) | 2026-08-29 01:58:38 Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 960010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 960010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 964633:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 964633:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 964633:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 964633:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 967236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 967236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 970665:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 970665:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 975287:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 975287:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 975287:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 978082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 978082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 981512:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 981512:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 986134:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 986134:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 986134:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 986134:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 989134:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 989134:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 993334:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 993334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 999108:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 999108:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 999108:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 999215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 999215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1003412:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1003412:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers= Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffer Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffer Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffer Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffer libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffer Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1008802:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1008802:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1008802:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1008802:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1009005:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1009005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 420 minutes (lustre-reviews_review-ldiskfs_130515.24) | 2026-08-29 02:03:39 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1013253:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1013253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1019027:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1019027:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1019027:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1019902:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1019902:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.28.182@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 1026073:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1026073:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1030021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1030021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1034643:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1034643:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1034643:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.28.182@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp | grep -E -A 1 '\s+nid: 10.240.27.230@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp3 | grep -E -A 1 '\s+nid: 10.240.27.230@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp | grep -E -A 1 '\s+nid: 10.240.27.230@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp | grep -E -A 1 '\s+nid: 10.240.27.230@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp | grep -E -A 1 '\s+nid: 10.240.27.230@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp3 | grep -E -A 1 '\s+nid: 10.240.27.230@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp | grep -E -A 1 '\s+nid: 10.240.27.230@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp3 | grep -E -A 1 '\s+nid: 10.240.27.230@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1036589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1036589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 LNetError: 1036589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1036589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.28.182@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1040019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1040019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.28.182@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.28.182@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1047130:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1047130:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1047130:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.28.151@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 1 previous similar message LNet: 1 peer NIs in recovery (showing 1): 10.240.28.151@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 154 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1048009:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1048009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 LNet: Removed LNI 10.240.28.182@tcp2 LNetError: 1048009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1048009:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 5 previous similar messages LNet: Removed LNI 10.240.28.182@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1051438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1051438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.28.182@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.28.182@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1058549:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1058549:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1058549:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.27.230@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.28.151@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.28.151@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 30 previous similar messages LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 128 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1059524:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1059524:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 LNet: Removed LNI 10.240.28.182@tcp2 LNet: Removed LNI 10.240.28.182@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing unload_modules_local LNet: 1062932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1062932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Autotest: Test running for 425 minutes (lustre-reviews_review-ldiskfs_130515.24) | 2026-08-29 02:08:39 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1066825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1066825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1071447:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1071447:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1071447:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.27.230@tcp -d 10.240.28.151@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message LNet: *** cfs_fail_loc=e002, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.28.151@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1073283:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1073283:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp LNet: Removed LNI 10.240.28.182@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1077480:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1077480:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.28.182@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.28.182@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-156vm227.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-156vm227.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.28.182@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm173.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm173.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1083254:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1083254:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1083254:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm182.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-157vm102.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-157vm182.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-157vm102.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-156vm227 Lustre: DEBUG MARKER: Disable routing on onyx-156vm227 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test (2026-08-28 22:21) |
| sanity-lnet test 100: Add route with single gw | LNetError: 1097685:0:(lib-move.c:1047:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 1097685:0:(lib-move.c:1047:lnet_return_rx_credits_locked()) LBUG CPU: 0 PID: 1097685 Comm: monitor_thread Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.137.1.el8_10_lustre.x86_64 #1 Hardware name: Red Hat KVM/RHEL, BIOS 1.16.3-4.el9 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.7+0x5/0x43 [libcfs] lnet_return_rx_credits_locked+0x3ac/0x550 [lnet] ? kvm_clock_get_cycles+0xd/0x20 ? ktime_get+0x3e/0xa0 lnet_msg_decommit+0x2e9/0x7c0 [lnet] lnet_finalize+0x3c6/0xb40 [lnet] ? lnet_ni_recv+0xef/0x330 [lnet] lnet_drop_routed_msgs_locked+0xac/0xe0 [lnet] ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_rtrpool_free_bufs+0xaa/0x160 [lnet] ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_rtrpools_free+0x52/0x90 [lnet] lnet_check_routers+0x5f6/0xad0 [lnet] ? wait_for_completion_interruptible_timeout+0xa6/0x130 ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] lnet_monitor_thread+0x4f7/0x670 [lnet] ? update_load_avg+0x7e/0x710 ? update_load_avg+0x7e/0x710 ? set_next_entity+0xb5/0x1e0 ? __switch_to_asm+0x43/0x80 ? finish_task_switch+0x86/0x2f0 ? __schedule+0x2d9/0x870 ? lnet_recover_local_nis.constprop.35+0x740/0x740 [lnet] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 863757:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 863757:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 395 minutes (lustre-reviews_review-ldiskfs-ubuntu_129208.86) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 866998:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 866998:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 871961:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 871961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNetError: 871961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 871961:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 875755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 875755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 880528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 880528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNetError: 880528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 880528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 884577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 884577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 889339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 889339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 892876:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 892876:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 897638:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 897638:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 902199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 902199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 906962:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 906962:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 914959:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 914959:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 400 minutes (lustre-reviews_review-ldiskfs-ubuntu_129208.86) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.41@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 920587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 920587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 923158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 923158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.41@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 928733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 928733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 931506:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 931506:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.41@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 936826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 936826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-75vm12.onyx.whamcloud.com to discover 10.240.30.41@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-75vm12.onyx.whamcloud.com to discover 10.240.30.41@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-124vm10 to discover 10.240.26.87@tcp Lustre: DEBUG MARKER: Force onyx-124vm10 to discover 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1065307 Lustre: DEBUG MARKER: Wait for 1065307 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1065307 Lustre: DEBUG MARKER: Finished wait on 1065307 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 940717:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 940717:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 946022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 946022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 950773:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 950773:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 950773:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 951693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 951693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 405 minutes (lustre-reviews_review-ldiskfs-ubuntu_129208.86) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 955281:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 955281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 960032:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 960032:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 960032:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 960032:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 960144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 960144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 963681:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 963681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 968431:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 968431:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 968431:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 971169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 971169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 974707:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 974707:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 979458:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 979458:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 979458:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 979458:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 982196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 982196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 985733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 985733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 990485:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 990485:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 990485:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 993021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 993021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 996558:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 996558:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1001310:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1001310:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1001310:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1001310:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1004048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1004048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Autotest: Test running for 410 minutes (lustre-reviews_review-ldiskfs-ubuntu_129208.86) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1008445:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1008445:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1014408:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1014408:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1014408:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1014520:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1014520:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1018865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1018865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1024424:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1024424:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1024424:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 1024424:0:(router.c:1707:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1024637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1024637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNetError: 1024637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1024637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1028982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1028982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1034945:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1034945:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1034945:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1035865:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1035865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNetError: 1035865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1035865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.41@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 1042180:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1042180:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1046259:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1046259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 415 minutes (lustre-reviews_review-ldiskfs-ubuntu_129208.86) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1051062:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1051062:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1051062:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.41@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp | grep -E -A 1 '\s+nid: 10.240.26.87@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp3 | grep -E -A 1 '\s+nid: 10.240.26.87@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp | grep -E -A 1 '\s+nid: 10.240.26.87@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp | grep -E -A 1 '\s+nid: 10.240.26.87@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp | grep -E -A 1 '\s+nid: 10.240.26.87@tcp$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp3 | grep -E -A 1 '\s+nid: 10.240.26.87@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp | grep -E -A 1 '\s+nid: 10.240.26.87@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp3 | grep -E -A 1 '\s+nid: 10.240.26.87@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1053069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1053069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 LNetError: 1053069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1053069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.41@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1056607:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1056607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.41@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.41@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1063899:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1063899:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1063899:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.27.80@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** LNet: 1 peer NIs in recovery (showing 1): 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1064823:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1064823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 LNetError: 1064823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1064823:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.41@tcp2 LNet: Removed LNI 10.240.30.41@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1068360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1068360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.41@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.41@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1075652:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1075652:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1075652:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.26.87@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.27.80@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.80@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.80@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 200 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1076677:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1076677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 LNetError: 1076677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 1076677:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.41@tcp2 LNet: Removed LNI 10.240.30.41@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing unload_modules_local LNet: 1080189:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1080189:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1084160:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1084160:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1088911:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1088911:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1088911:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.26.87@tcp -d 10.240.27.80@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.27.80@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1090842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1090842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp LNet: Removed LNI 10.240.30.41@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Autotest: Test running for 420 minutes (lustre-reviews_review-ldiskfs-ubuntu_129208.86) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 1095238:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 1095238:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.41@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-142vm27.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-142vm27.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.41@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-124vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-124vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.41@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 1101201:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 1101201:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 1101201:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-75vm12.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-75vm12.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-124vm10 Lustre: DEBUG MARKER: Disable routing on onyx-124vm10 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test (2026-08-13 18:57) |
| sanity-lnet test 100: Add route with single gw | LNetError: 859408:0:(lib-move.c:1211:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 859408:0:(lib-move.c:1211:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 859408 Comm: monitor_thread Kdump: loaded Tainted: G OE -------- - - 4.18.0-553.71.1.el8_lustre.x86_64 #1 Hardware name: Red Hat KVM, BIOS 1.16.0-4.module+el8.8.0+1454+0b2cbfb8 04/01/2014 Call Trace: dump_stack+0x41/0x60 lbug_with_loc.cold.6+0x5/0x43 [libcfs] lnet_return_rx_credits_locked+0x3c7/0x570 [lnet] ? cfs_percpt_unlock+0x30/0xb0 [lnet] lnet_msg_decommit+0x288/0x760 [lnet] lnet_finalize+0x3b7/0xa80 [lnet] ? _raw_spin_unlock_bh+0xa/0x20 ? ksocknal_recv+0xc1/0x2a0 [ksocklnd] lnet_drop_routed_msgs_locked+0xac/0xe0 [lnet] ? lnet_recover_local_nis.constprop.39+0x740/0x740 [lnet] lnet_rtrpool_free_bufs+0xaa/0x160 [lnet] ? lnet_recover_local_nis.constprop.39+0x740/0x740 [lnet] lnet_rtrpools_free+0x52/0x90 [lnet] lnet_check_routers+0x5cc/0xab0 [lnet] ? wait_for_completion_interruptible_timeout+0xa6/0x130 ? lnet_recover_local_nis.constprop.39+0x740/0x740 [lnet] lnet_monitor_thread+0x4f7/0x670 [lnet] ? update_load_avg+0x7e/0x710 ? newidle_balance+0x2b6/0x3b0 ? __switch_to_asm+0x43/0x80 ? finish_task_switch+0x86/0x2f0 ? __schedule+0x2d9/0x870 ? lnet_recover_local_nis.constprop.39+0x740/0x740 [lnet] kthread+0x134/0x150 ? set_kthread_struct+0x50/0x50 ret_from_fork+0x35/0x40 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 732858:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 732858:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 734712:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 734712:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 737472:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 737472:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 739900:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 739900:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 742660:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 742660:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 745333:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 745333:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 748093:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 748093:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 750275:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 750275:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 753035:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 753035:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 756198:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 756198:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 758958:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 758958:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 762488:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 762488:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.29.130@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 766307:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 766307:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 768169:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 768169:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.29.130@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 771987:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 771987:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 235 minutes (lustre-reviews_review-ldiskfs-arm_116878.51) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 774150:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 774150:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.29.130@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 777722:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 777722:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm1.onyx.whamcloud.com to discover 10.240.29.130@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm1.onyx.whamcloud.com to discover 10.240.29.130@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-116vm3 to discover 10.240.30.112@tcp Lustre: DEBUG MARKER: Force onyx-116vm3 to discover 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1169635 Lustre: DEBUG MARKER: Wait for 1169635 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1169635 Lustre: DEBUG MARKER: Finished wait on 1169635 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 780843:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 780843:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 784735:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 784735:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 787491:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 787491:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 787491:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 787786:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 787786:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 789970:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 789970:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 792723:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 792723:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 792723:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 792723:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 792826:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 792826:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 795008:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 795008:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 797761:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 797761:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 797761:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 800745:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 800745:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 802927:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 802927:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 805680:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 805680:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 805680:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 805680:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Autotest: Test running for 240 minutes (lustre-reviews_review-ldiskfs-arm_116878.51) Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 808912:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 808912:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 811094:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 811094:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 813847:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 813847:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 813847:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 816830:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 816830:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 819012:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 819012:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 821766:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 821766:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 821766:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 821766:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 824749:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 824749:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 827126:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 827126:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 830167:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 830167:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 830167:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 830271:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 830271:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 832646:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 832646:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 835591:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 835591:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 835591:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 835591:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.30.113@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 835791:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 835791:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 838165:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 838165:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 841206:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 841206:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 841206:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 841502:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 841502:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.29.130@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 846017:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 846017:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing unload_modules_local LNet: 848562:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 848562:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 245 minutes (lustre-reviews_review-ldiskfs-arm_116878.51) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 851287:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 851287:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 854040:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 854040:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 854040:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.30.113@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.112@tcp -d 10.240.30.113@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.30.113@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 855872:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 855872:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp LNet: Removed LNI 10.240.29.130@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 858247:0:(lib-ptl.c:966:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 858247:0:(acceptor.c:263:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.29.130@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.29.130@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-116vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-116vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.29.130@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 861288:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 861288:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 861288:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-70vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-70vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-116vm3 Lustre: DEBUG MARKER: Disable routing on onyx-116vm3 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test (2025-09-24 10:00) |