| Match messages in logs (every line would be required to be present in log output Copy from "Messages before crash" column below): | |
| Match messages in full crash (every line would be required to be present in crash log output Copy from "Full Crash" column below): | |
| Limit to a test: (Copy from below "Failing text"): | |
| Delete these reports as invalid (real bug in review or some such) | |
| Bug or comment: | |
| Extra info: |
| Failing Test | Full Crash | Messages before crash | Comment |
|---|---|---|---|
| sanity-lnet test 100: Add route with single gw | LNetError: 439839:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 439839:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) LBUG CPU: 0 PID: 439839 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 317482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 317482:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 319142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 319142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 321842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 321842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 323995:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 323995:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 326494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 326494:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 328856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 328856:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 331355:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 331355:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 333297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 333297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Autotest: Test running for 255 minutes (lustre-reviews_review-ldiskfs-arm_122952.60) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 335841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 335841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 338623:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 338623:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 341122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 341122:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 346736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 346736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 350185:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 350185:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 351841:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 351841:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 355286:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 355286:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 357154:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 357154:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 360387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 360387:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm1.onyx.whamcloud.com to discover 10.240.30.113@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm1.onyx.whamcloud.com to discover 10.240.30.113@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm2 to discover 10.240.30.112@tcp Lustre: DEBUG MARKER: Force onyx-154vm2 to discover 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1224785 Lustre: DEBUG MARKER: Wait for 1224785 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1224785 Lustre: DEBUG MARKER: Finished wait on 1224785 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 363121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 363121:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 366524:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 366524:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 369016:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 369016:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 369016:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Autotest: Test running for 260 minutes (lustre-reviews_review-ldiskfs-arm_122952.60) Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 369315:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 369315:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 371256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 371256:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 373748:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 373748:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 373748:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 373748:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 373838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 373838:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 375779:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 375779:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 378271:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 378271:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 378271:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 380984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 380984:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 382926:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 382926:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 385418:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 385418:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 385418:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 385418:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 388295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 388295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 390236:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 390236:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 392728:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 392728:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 392728:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 395442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 395442:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 397384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 397384:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 399876:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 399876:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 399876:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 399876:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 402590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 402590:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 404699:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 404699:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 407437:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 407437:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 407437:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 407526:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 407526:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 265 minutes (lustre-reviews_review-ldiskfs-arm_122952.60) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 409676:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 409676:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 412333:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 412333:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 412333:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 412333:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.148@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 412504:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 412504:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 414610:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 414610:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 417348:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 417348:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 417348:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 417601:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 417601:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 421643:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 421643:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 424040:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 424040:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 426532:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 426532:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 426532:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.113@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.112@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.112@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.112@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 427877:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 427877:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 LNet: Removed LNI 10.240.30.113@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 430295:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 430295:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 432644:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 432644:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 435137:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 435137:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 435137:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.148@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.112@tcp -d 10.240.26.148@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.148@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 436703:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 436703:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 438809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 438809:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm13.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm13.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 441547:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 441547:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 441547:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm14.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm14.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-154vm2 Lustre: DEBUG MARKER: Disable routing on onyx-154vm2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish Autotest: Test running for 270 minutes (lustre-reviews_review-ldiskfs-arm_122952.60) | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 451175:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 451175:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) LBUG CPU: 0 PID: 451175 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 315578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 315578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 317244:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 317244:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 319953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 319953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 322111:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 322111:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 324619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 324619:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 326988:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 326988:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 329497:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 329497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 331444:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 331444:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 333953:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 333953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 336741:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 336741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 339249:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 339249:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 344869:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 344869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.125@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Autotest: Test running for 265 minutes (lustre-reviews_review-ldiskfs-arm_122925.60) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 348369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 348369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 350034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 350034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.125@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 353494:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 353494:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 355368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 355368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.125@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 358613:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 358613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm13.onyx.whamcloud.com to discover 10.240.30.125@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm13.onyx.whamcloud.com to discover 10.240.30.125@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm14 to discover 10.240.30.124@tcp Lustre: DEBUG MARKER: Force onyx-154vm14 to discover 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1213327 Lustre: DEBUG MARKER: Wait for 1213327 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1213327 Lustre: DEBUG MARKER: Finished wait on 1213327 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 361353:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 361353:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 364762:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 364762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 367263:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 367263:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 367263:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 367518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 367518:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 369465:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 369465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 371966:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 371966:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 371966:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 371966:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 372055:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 372055:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 374002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 374002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 376503:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 376503:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 376503:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 379052:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 379052:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 380999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 380999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 383500:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 383500:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 383500:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 383500:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Autotest: Test running for 270 minutes (lustre-reviews_review-ldiskfs-arm_122925.60) Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 386259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 386259:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 388206:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 388206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 390707:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 390707:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 390707:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 393421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 393421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 395368:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 395368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 397869:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 397869:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 397869:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 397869:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 400582:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 400582:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 402697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 402697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 405444:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 405444:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 405444:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 405533:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 405533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 407645:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 407645:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 410310:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 410310:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 410310:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 410310:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 410482:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 410482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 412594:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 412594:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 415342:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 415342:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 415342:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 415596:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 415596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.125@tcp1 Autotest: Test running for 275 minutes (lustre-reviews_review-ldiskfs-arm_122925.60) Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 419697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 419697:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 422101:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 422101:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 424602:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 424602:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 424602:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.125@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp | grep -E -A 1 '\s+nid: 10.240.30.124@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.124@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp | grep -E -A 1 '\s+nid: 10.240.30.124@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.124@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp | grep -E -A 1 '\s+nid: 10.240.30.124@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.124@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 425865:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 425865:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 LNet: Removed LNI 10.240.30.125@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 427812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 427812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.125@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.125@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 431734:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 431734:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 431734:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.143@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 199 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 432484:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 432484:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 LNet: Removed LNI 10.240.30.125@tcp2 LNet: Removed LNI 10.240.30.125@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 434431:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 434431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.125@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.125@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 438353:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 438353:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 438353:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.124@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.143@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 199 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 439184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 439184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 LNet: Removed LNI 10.240.30.125@tcp2 LNet: Removed LNI 10.240.30.125@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing unload_modules_local LNet: 441608:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 441608:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 443963:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 443963:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 446464:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 446464:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 446464:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.124@tcp -d 10.240.26.143@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 11 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 448030:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 448030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp LNet: Removed LNI 10.240.30.125@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 450142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 450142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.125@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.125@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm14.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm14.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.125@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 452889:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 452889:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 452889:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm13.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm13.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-154vm14 Lustre: DEBUG MARKER: Disable routing on onyx-154vm14 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 423885:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 423885:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 423885 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20241117-4.el9 11/17/2024 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 310261:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 310261:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 311916:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 311916:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 314401:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 314401:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 316547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 316547:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 319031:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 319031:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 321387:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 321387:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 323872:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 323872:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 325807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 325807:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 328290:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 328290:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 331065:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 331065:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 333550:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 333550:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 338108:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 338108:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.39@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 341538:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 341538:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 343190:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 343190:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.39@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 346619:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 346619:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 348481:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 348481:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.39@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 351697:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 351697:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 280 minutes (lustre-reviews_review-ldiskfs-arm_122716.61) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.22.38@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-180vm21.onyx.whamcloud.com to discover 10.240.22.39@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-180vm21.onyx.whamcloud.com to discover 10.240.22.39@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-180vm22 to discover 10.240.22.38@tcp Lustre: DEBUG MARKER: Force onyx-180vm22 to discover 10.240.22.38@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.22.38@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1178513 Lustre: DEBUG MARKER: Wait for 1178513 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1178513 Lustre: DEBUG MARKER: Finished wait on 1178513 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 354471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 354471:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 357867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 357867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 360345:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 360345:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 360345:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 360599:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 360599:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 362535:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 362535:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 365011:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 365011:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 365011:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 365011:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 365100:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 365100:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 367036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 367036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 369513:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 369513:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 369513:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.22.38@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 372063:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 372063:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 373998:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 373998:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 376475:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 376475:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 376475:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 376475:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 379025:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 379025:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 380960:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 380960:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 383436:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 383436:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 383436:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.22.38@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 385986:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 385986:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 387923:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 387923:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 390400:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 390400:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 390400:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 390400:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 392949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 392949:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 395054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 395054:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 397777:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 397777:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 397777:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 397867:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 397867:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 399966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 399966:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 402606:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 402606:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 402606:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 402606:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Autotest: Test running for 285 minutes (lustre-reviews_review-ldiskfs-arm_122716.61) LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 402823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 402823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 404922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 404922:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 407646:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 407646:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 407646:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 407899:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 407899:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.22.39@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 411924:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 411924:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing unload_modules_local LNet: 414370:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 414370:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 416713:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 416713:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 419190:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 419190:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 419190:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.22.38@tcp -d 10.240.26.143@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.143@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 420757:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 420757:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp LNet: Removed LNI 10.240.22.39@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 422858:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 422858:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.22.39@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.22.39@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm22.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm22.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.22.39@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 425581:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 425581:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 425581:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Autotest: Test running for 290 minutes (lustre-reviews_review-ldiskfs-arm_122716.61) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm21.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm9.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm21.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm9.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-180vm22 Lustre: DEBUG MARKER: Disable routing on onyx-180vm22 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 147682:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 147682:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 147682 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 12121:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 12121:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 13771:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 13771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 16471:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 16471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 18624:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 18624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 21123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 21123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 23485:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 23485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 25984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 25984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 27925:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 27925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Autotest: Test running for 5 minutes (lustre-reviews_review-ldiskfs-arm_122722.61) Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 30470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 30470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 33253:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 33253:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 35752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 35752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 41367:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 41367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.117@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 44812:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 44812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 46470:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 46470:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.117@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 49914:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 49914:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 51783:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 51783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.117@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 55016:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 55016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm5.onyx.whamcloud.com to discover 10.240.30.117@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm5.onyx.whamcloud.com to discover 10.240.30.117@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm6 to discover 10.240.30.116@tcp Lustre: DEBUG MARKER: Force onyx-154vm6 to discover 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 91457 Lustre: DEBUG MARKER: Wait for 91457 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 91457 Lustre: DEBUG MARKER: Finished wait on 91457 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 57752:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 57752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 10 minutes (lustre-reviews_review-ldiskfs-arm_122722.61) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 61198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 61198:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 63692:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 63692:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 63692:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 63947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 63947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 65888:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 65888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 68380:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 68380:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 68380:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 68380:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 68469:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 68469:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 70410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 70410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 72902:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 72902:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 72902:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 75452:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 75452:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 77393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 77393:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 79885:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 79885:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 79885:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 79885:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 82598:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 82598:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 84540:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 84540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 87032:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 87032:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 87032:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 89745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 89745:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 91689:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 91689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 94181:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 94181:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 94181:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 94181:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 96895:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 96895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 99003:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 99003:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 101741:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 101741:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 101741:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Autotest: Test running for 15 minutes (lustre-reviews_review-ldiskfs-arm_122722.61) LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 101874:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 101874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 103982:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 103982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 106638:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 106638:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 106638:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 106638:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 106810:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 106810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 108915:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 108915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 111653:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 111653:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 111653:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNetError: 110655:0:(socklnd.c:1543:ksocknal_destroy_conn()) Completing partial receive from 12345-10.240.26.146@tcp1[2], ip 10.240.26.146:1022, with error, wanted: 1048576, left: 1048576, last alive is 103 secs ago LNet: 111908:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 111908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.117@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 116161:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 116161:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 118559:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 118559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 121051:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 121051:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 121051:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.117@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp | grep -E -A 1 '\s+nid: 10.240.30.116@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.116@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp | grep -E -A 1 '\s+nid: 10.240.30.116@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp | grep -E -A 1 '\s+nid: 10.240.30.116@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.116@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp | grep -E -A 1 '\s+nid: 10.240.30.116@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.116@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 122396:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 122396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 LNetError: 122396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 122396:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.117@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 124338:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 124338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.117@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Autotest: Test running for 20 minutes (lustre-reviews_review-ldiskfs-arm_122722.61) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.117@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 128290:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 128290:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 128290:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.145@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.145@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 84 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 129039:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 129039:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 LNet: Removed LNI 10.240.30.117@tcp2 LNet: Removed LNI 10.240.30.117@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 130981:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 130981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.117@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.117@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 134888:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 134888:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 134888:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.116@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.145@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.145@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 199 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 135720:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 135720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 LNet: Removed LNI 10.240.30.117@tcp2 LNet: Removed LNI 10.240.30.117@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing unload_modules_local LNet: 138139:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 138139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 140488:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 140488:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 142980:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 142980:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 142980:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.116@tcp -d 10.240.26.145@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 11 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.145@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 144547:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 144547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp LNet: Removed LNI 10.240.30.117@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 146652:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 146652:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.117@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm10.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm10.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.117@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm6.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm6.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.117@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 149390:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 149390:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 149390:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm5.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm5.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-154vm6 Lustre: DEBUG MARKER: Disable routing on onyx-154vm6 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 451930:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 451930:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) LBUG CPU: 0 PID: 451930 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 316245:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 316245:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 317906:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 317906:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Autotest: Test running for 355 minutes (lustre-reviews_review-ldiskfs-arm_122443.61) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 320649:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 320649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 322801:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 322801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 325300:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 325300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 327663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 327663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 330162:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 330162:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 332103:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 332103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 334603:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 334603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 337386:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 337386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 339885:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 339885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 345499:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 345499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.119@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 348949:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 348949:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 350607:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 350607:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.119@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 354053:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 354053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 355919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 355919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.119@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 359152:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 359152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 360 minutes (lustre-reviews_review-ldiskfs-arm_122443.61) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm7.onyx.whamcloud.com to discover 10.240.30.119@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm7.onyx.whamcloud.com to discover 10.240.30.119@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm8 to discover 10.240.30.118@tcp Lustre: DEBUG MARKER: Force onyx-154vm8 to discover 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1222178 Lustre: DEBUG MARKER: Wait for 1222178 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1222178 Lustre: DEBUG MARKER: Finished wait on 1222178 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 361933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 361933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 365334:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 365334:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 367826:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 367826:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 367826:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 368080:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 368080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 370022:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 370022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 372515:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 372515:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 372515:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 372515:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 372604:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 372604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 374545:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 374545:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 377037:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 377037:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 377037:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 379751:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 379751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 381692:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 381692:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 384184:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 384184:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 384184:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 384184:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 386897:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 386897:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 388839:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 388839:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 391331:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 391331:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 391331:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 394208:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 394208:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 396148:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 396148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 398639:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 398639:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 398639:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 398639:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Autotest: Test running for 365 minutes (lustre-reviews_review-ldiskfs-arm_122443.61) Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 401397:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 401397:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 403507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 403507:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 406245:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 406245:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 406245:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 406335:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 406335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 408442:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 408442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 411099:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 411099:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 411099:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 411099:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 411271:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 411271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 413377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 413377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 416115:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 416115:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 416115:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 416369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 416369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.119@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 420411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 420411:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 422805:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 422805:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 425297:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 425297:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 425297:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.119@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp | grep -E -A 1 '\s+nid: 10.240.30.118@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.118@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp | grep -E -A 1 '\s+nid: 10.240.30.118@tcp$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp | grep -E -A 1 '\s+nid: 10.240.30.118@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.118@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp | grep -E -A 1 '\s+nid: 10.240.30.118@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.118@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 426642:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 426642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 LNetError: 426642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 426642:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.119@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 370 minutes (lustre-reviews_review-ldiskfs-arm_122443.61) Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 428628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 428628:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.119@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.119@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 432536:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 432536:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 432536:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: 1 peer NIs in recovery (showing 1): 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 433285:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 433285:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 LNet: Removed LNI 10.240.30.119@tcp2 LNet: Removed LNI 10.240.30.119@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 435228:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 435228:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.119@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.119@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 439135:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 439135:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 439135:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.118@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 1 previous similar message LNet: 1 peer NIs in recovery (showing 1): 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 439967:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 439967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 LNet: Removed LNI 10.240.30.119@tcp2 LNet: Removed LNI 10.240.30.119@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing unload_modules_local LNet: 442384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 442384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 444735:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 444735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 447227:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 447227:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 447227:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.118@tcp -d 10.240.26.138@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 11 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.138@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 448794:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 448794:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp LNet: Removed LNI 10.240.30.119@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 450900:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 450900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.119@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm3.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm3.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.119@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm8.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm8.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.119@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 453638:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 453638:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 453638:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm7.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm7.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-154vm8 Lustre: DEBUG MARKER: Disable routing on onyx-154vm8 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 493297:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 493297:0:(lib-move.c:1215:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 493297 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 356527:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 356527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 358211:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 358211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 360947:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 360947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 363123:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 363123:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 365659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 365659:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 368046:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 368046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 370581:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 370581:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 372546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 372546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 375081:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 375081:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 377903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 377903:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 380438:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 380438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 290 minutes (lustre-reviews_review-ldiskfs-dne-arm_122395.74) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 386122:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 386122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.141@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 389631:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 389631:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 391313:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 391313:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.141@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 394817:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 394817:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 396709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 396709:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.141@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 400002:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 400002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm29.onyx.whamcloud.com to discover 10.240.30.141@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm29.onyx.whamcloud.com to discover 10.240.30.141@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm30 to discover 10.240.30.140@tcp Lustre: DEBUG MARKER: Force onyx-154vm30 to discover 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1448817 Lustre: DEBUG MARKER: Wait for 1448817 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1448817 Lustre: DEBUG MARKER: Finished wait on 1448817 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 402758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 402758:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 406184:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 406184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 408712:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 408712:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 408712:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 408966:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 408966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 410931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 410931:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 413459:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 413459:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 413459:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 413459:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 413548:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 413548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 415513:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 415513:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 295 minutes (lustre-reviews_review-ldiskfs-dne-arm_122395.74) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 418088:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 418088:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 418088:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 420637:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 420637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 422602:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 422602:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 425130:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 425130:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 425130:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 425130:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 427844:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 427844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 429809:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 429809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 432338:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 432338:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 432338:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 435051:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 435051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 437017:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 437017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 439545:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 439545:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 439545:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 439545:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 442258:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 442258:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 444392:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 444392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 447166:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 447166:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 447166:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 447256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 447256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 449384:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 449384:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 452076:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 452076:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 452076:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 452076:0:(router.c:1701:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 452248:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 452248:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 454377:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 454377:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 457151:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 457151:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 457151:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Autotest: Test running for 300 minutes (lustre-reviews_review-ldiskfs-dne-arm_122395.74) Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 457451:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 457451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.141@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 461553:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 461553:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 463974:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 463974:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 466503:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 466503:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 466503:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.141@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp | grep -E -A 1 '\s+nid: 10.240.30.140@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.140@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp | grep -E -A 1 '\s+nid: 10.240.30.140@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.140@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp | grep -E -A 1 '\s+nid: 10.240.30.140@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.140@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 467775:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 467775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 LNetError: 467775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 467775:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.141@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 469740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 469740:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.141@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.141@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 473706:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 473706:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 473706:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.139@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 199 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 474455:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 474455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 LNetError: 474455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 474455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.141@tcp2 LNet: Removed LNI 10.240.30.141@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 476421:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 476421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp2 --if eth0 LNet: Added LNI 10.240.30.141@tcp2 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.141@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 480388:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 480388:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 480388:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.140@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe004 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set health_sensitivity 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer set --health 0 --nid 10.240.26.139@tcp1 LNet: *** cfs_fail_loc=e004, val=0*** Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health LNet: *** cfs_fail_loc=e004, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.139@tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 | grep -e nid -e health Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 481219:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 481219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 LNetError: 481219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNetError: 481219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages LNet: Removed LNI 10.240.30.141@tcp2 LNet: Removed LNI 10.240.30.141@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing unload_modules_local LNet: 483660:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 483660:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 486034:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 486034:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 488562:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 488562:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 488562:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.140@tcp -d 10.240.26.139@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.139@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 490128:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 490128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp LNet: Removed LNI 10.240.30.141@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 492256:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 492256:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.141@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.141@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm30.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm30.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.141@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm4.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm4.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 495029:0:(router.c:1476:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 495029:0:(router.c:1498:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 495029:0:(router.c:1520:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm29.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm29.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm5.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm5.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-154vm30 Lustre: DEBUG MARKER: Disable routing on onyx-154vm30 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 423683:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 423683:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 423683 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS edk2-20240524-6.el9_5.3 05/24/2024 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 309056:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 309056:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 310709:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 310709:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 313400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 313400:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 315546:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 315546:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 318036:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 318036:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 320393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 320393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 322883:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 322883:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 324818:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 324818:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 327308:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 327308:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 330085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 330085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 332575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 332575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 337133:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 337133:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.25.42@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 340567:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 340567:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 342217:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 342217:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Autotest: Test running for 225 minutes (lustre-reviews_review-ldiskfs-arm_119982.52) Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.25.42@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 345690:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 345690:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 347552:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 347552:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.25.42@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 350770:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 350770:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.41@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-180vm1.onyx.whamcloud.com to discover 10.240.25.42@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-180vm1.onyx.whamcloud.com to discover 10.240.25.42@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-180vm2 to discover 10.240.25.41@tcp Lustre: DEBUG MARKER: Force onyx-180vm2 to discover 10.240.25.41@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.25.41@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1180815 Lustre: DEBUG MARKER: Wait for 1180815 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1180815 Lustre: DEBUG MARKER: Finished wait on 1180815 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 353498:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 353498:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 356891:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 356891:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 359374:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 359374:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 359374:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 359628:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 359628:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 361563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 361563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 364047:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 364047:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 364047:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 364047:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 364136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 364136:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 366071:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 366071:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 368555:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 368555:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 368555:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.25.41@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 371269:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 371269:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 373204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 373204:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 375687:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 375687:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 375687:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 375687:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Autotest: Test running for 230 minutes (lustre-reviews_review-ldiskfs-arm_119982.52) Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 378446:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 378446:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 380382:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 380382:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 382865:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 382865:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 382865:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.25.41@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 385578:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 385578:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 387514:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 387514:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 389996:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 389996:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 389996:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 389996:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 392724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 392724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 394827:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 394827:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 397557:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 397557:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 397557:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 397646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 397646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 399745:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 399745:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 402391:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 402391:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 402391:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 402391:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.136@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 402563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 402563:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 404663:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 404663:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 407392:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 407392:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 407392:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 407646:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 407646:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.25.42@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 411673:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 411673:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 235 minutes (lustre-reviews_review-ldiskfs-arm_119982.52) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 414163:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 414163:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 416507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 416507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 418991:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 418991:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 418991:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.136@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.25.41@tcp -d 10.240.26.136@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 8 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 2 previous similar messages Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.136@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 420557:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 420557:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp LNet: Removed LNI 10.240.25.42@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 422656:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 422656:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.25.42@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.25.42@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-180vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.25.42@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 425385:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 425385:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 425385:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-180vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-180vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-180vm2 Lustre: DEBUG MARKER: Disable routing on onyx-180vm2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |
| sanity-lnet test 100: Add route with single gw | LNetError: 432957:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) ASSERTION( msg->msg_kiov != ((void *)0) ) failed: LNetError: 432957:0:(lib-move.c:1213:lnet_return_rx_credits_locked()) LBUG CPU: 1 PID: 432957 Comm: monitor_thread Kdump: loaded Tainted: G OE ------- --- 5.14.0-503.40.1.el9_5.aarch64+64k #1 Hardware name: QEMU KVM Virtual Machine, BIOS 0.0.0 02/06/2015 Call trace: dump_backtrace+0xa8/0x120 show_stack+0x1c/0x30 dump_stack_lvl+0x74/0x8c dump_stack+0x14/0x24 lbug_with_loc+0x64/0xa0 [libcfs] lnet_return_rx_credits_locked+0x474/0x5e0 [lnet] lnet_msg_decommit_rx+0xd0/0x304 [lnet] lnet_msg_decommit+0x88/0x270 [lnet] lnet_complete_msg_locked+0x5c/0x2a0 [lnet] lnet_finalize+0x12c/0x2e0 [lnet] lnet_drop_routed_msgs_locked+0xbc/0xf0 [lnet] lnet_rtrpool_free_bufs+0xc4/0x170 [lnet] lnet_rtrpools_free.part.0+0x5c/0xa0 [lnet] lnet_check_routers+0x5f4/0x8cc [lnet] lnet_monitor_thread+0x1b0/0x1d0 [lnet] kthread+0xe8/0xf4 ret_from_fork+0x10/0x20 | Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod Key type lgssc unregistered LNet: 311015:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 311015:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 312669:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 312669:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 315359:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 315359:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 317506:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 317506:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 319996:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 319996:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 322352:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 322352:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 324842:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 324842:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 326777:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 326777:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 329267:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 329267:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 332045:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 332045:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 334534:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 334534:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 340142:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 340142:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 343575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 343575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 345226:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 345226:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 348658:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 348658:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 350521:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 350521:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 353736:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 353736:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 245 minutes (lustre-reviews_review-ldiskfs-arm_119041.51) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark Initial discovery Lustre: DEBUG MARKER: Initial discovery Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: Fail local discover ping to set LNET_PEER_REDISCOVER flag Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm1.onyx.whamcloud.com to discover 10.240.30.113@tcp \(in background\) Lustre: DEBUG MARKER: Force onyx-154vm1.onyx.whamcloud.com to discover 10.240.30.113@tcp (in background) Lustre: DEBUG MARKER: /usr/sbin/lctl mark Force onyx-154vm2 to discover 10.240.30.112@tcp Lustre: DEBUG MARKER: Force onyx-154vm2 to discover 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl discover --force 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for 1192183 Lustre: DEBUG MARKER: Wait for 1192183 Lustre: DEBUG MARKER: /usr/sbin/lctl mark Finished wait on 1192183 Lustre: DEBUG MARKER: Finished wait on 1192183 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 356507:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 356507:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 359903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 359903:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 362386:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 362386:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 362386:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 362641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 362641:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 364575:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 364575:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 367061:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 367061:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 367061:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 367061:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 367150:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 367150:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 369085:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 369085:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 371583:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 371583:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 371583:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 374297:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 374297:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 376232:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 376232:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=1 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 378715:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 378715:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 378715:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 378715:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 381429:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 381429:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 383364:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 383364:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 385847:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 385847:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 385847:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 388724:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 388724:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 250 minutes (lustre-reviews_review-ldiskfs-arm_119041.51) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 390704:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 390704:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet avoid_asym_router_failure=0 lnet_peer_discovery_disabled=1 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 393188:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 393188:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 393188:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 393188:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 395901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 395901:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 398004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 398004:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 400734:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 400734:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 400734:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 400823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 400823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 402922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 402922:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 t libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet lnet_peer_discovery_disabled=1 lnet_health_sensitivity=0 lnet_transaction_timeout=10 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers= Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 405568:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 405568:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 405568:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: 405568:0:(router.c:1706:lnet_rtrpools_enable()) Consider turning discovery on to enable full Multi-Rail routing functionality LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.136@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 405740:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 405740:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 407840:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 407840:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 410569:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 410569:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 410569:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 410823:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 410823:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net del --net tcp1 --if eth0 LNet: Removed LNI 10.240.30.113@tcp1 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 414851:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 414851:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 417242:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 417242:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 419725:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 419725:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 419725:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp3 --if eth0 LNet: Added LNI 10.240.30.113@tcp3 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.112@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.112@tcp3$' | awk '/\s+state:/{print $NF}' LNet: 1 peer NIs in recovery (showing 1): 10.240.30.112@tcp Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp | grep -E -A 1 '\s+nid: 10.240.30.112@tcp$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show --nid 10.240.30.112@tcp3 | grep -E -A 1 '\s+nid: 10.240.30.112@tcp3$' | awk '/\s+state:/{print $NF}' Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 420983:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 420983:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 LNet: Removed LNI 10.240.30.113@tcp3 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing unload_modules_local LNet: 423393:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 423393:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Autotest: Test running for 255 minutes (lustre-reviews_review-ldiskfs-arm_119041.51) Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 425781:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 425781:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet peer_buffer_credits=1024 alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 428265:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 428265:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 428265:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v --nid 10.240.26.136@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show --net tcp1 --verbose Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 0 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 10 Lustre: DEBUG MARKER: /usr/sbin/lctl set_param fail_loc=0xe002 Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay add -s 10.240.30.112@tcp -d 10.240.26.136@tcp1 -l 11 -r 1 -m GET LNet: *** cfs_fail_loc=e002, val=0*** LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 11 previous similar messages LNet: *** cfs_fail_loc=e002, val=0*** LNet: Skipped 1 previous similar message Lustre: DEBUG MARKER: /usr/sbin/lctl net_delay del -a Lustre: DEBUG MARKER: /usr/sbin/lnetctl set retry_count 2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set -h Lustre: DEBUG MARKER: /usr/sbin/lnetctl global show Lustre: DEBUG MARKER: /usr/sbin/lnetctl net set --net tcp1 --lnd-timeout 50 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set transaction_timeout 151 Lustre: DEBUG MARKER: /usr/sbin/lnetctl peer show -v 2 --nid 10.240.26.136@tcp1 Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 429831:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 429831:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp LNet: Removed LNI 10.240.30.113@tcp1 Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure --all LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing lnet_if_list Lustre: DEBUG MARKER: /usr/sbin/lctl get_param -n cpu_partition_table | wc -l Lustre: DEBUG MARKER: /usr/sbin/lnetctl net show -v Lustre: DEBUG MARKER: /usr/sbin/lustre_rmmod LNet: 431930:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit LNetError: 431930:0:(acceptor.c:264:lnet_acceptor_remove_socket()) Interface eth0 not found LNet: Removed LNI 10.240.30.113@tcp Key type .llcrypt unregistered Key type ._llcrypt unregistered Key type ._llcrypt registered Key type .llcrypt registered Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 libcfs: HW NUMA nodes: 1, HW CPU cores: 2, npartitions: 1 alg: No test for adler32 (adler32-zlib) Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing load_lnet alive_router_check_interval=5 router_ping_timeout=5 large_router_buffers=4 small_router_buffers=8 tiny_router_buffers=9 Lustre: DEBUG MARKER: /usr/sbin/lnetctl lnet configure Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm1.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm1.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp --if eth0 LNet: Added LNI 10.240.30.113@tcp [8/256/0/180] Lustre: DEBUG MARKER: PATH=/usr/lib64/lustre/tests:/usr/lib/lustre/tests:/usr/lib64/lustre/tests:/opt/iozone/bin:/opt/iozone/bin:/opt/iozone/bin:/usr/lib64/lustre/tests/mpi:/usr/lib64/lustre/tests/racer:/usr/lib64/lustre/../lustre-iokit/sgpdd-survey:/usr/lib64/lustre/tests:/us Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm2.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-154vm2.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 LNet: Added LNI 10.240.30.113@tcp1 [8/256/0/180] Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing \/usr\/sbin\/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing /usr/sbin/lnetctl net add --net tcp1 --if eth0 Lustre: DEBUG MARKER: /usr/sbin/lctl list_nids Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 1 LNet: 434660:0:(router.c:1481:lnet_nrb_tiny_calculate()) tiny_router_buffers=9 less than recommended minimum 512 LNet: 434660:0:(router.c:1503:lnet_nrb_small_calculate()) small_router_buffers=8 less than recommended minimum 4096 LNet: 434660:0:(router.c:1525:lnet_nrb_large_calculate()) large_router_buffers=4 less than recommended minimum 256 LNet: Message forwarding will be enabled in 10s LNet: Message forwarding enabled Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-154vm1.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-154vm1.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: /usr/sbin/lctl mark onyx-78vm11.onyx.whamcloud.com: executing load_module ..\/lnet\/selftest\/lnet_selftest Lustre: DEBUG MARKER: onyx-78vm11.onyx.whamcloud.com: executing load_module ../lnet/selftest/lnet_selftest Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/alive_router_check_interval Lustre: DEBUG MARKER: echo 5 > /sys/module/lnet/parameters/router_ping_timeout Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait 5s for LST to start Lustre: DEBUG MARKER: Wait 5s for LST to start Lustre: DEBUG MARKER: /usr/sbin/lctl mark Disable routing on onyx-154vm2 Lustre: DEBUG MARKER: Disable routing on onyx-154vm2 Lustre: DEBUG MARKER: /usr/sbin/lnetctl set routing 0 LNet: Message forwarding will stop in 10s to 30s Lustre: DEBUG MARKER: /usr/sbin/lctl mark Wait for lst to finish Lustre: DEBUG MARKER: Wait for lst to finish | Link to test |