Argonne National Laboratory

GM/CA @ APS

Remote Access - Speed Tests

Department of Energy Office of Science
GM/CA @ APS Sponsors:
National Institute of General Medical Sciences (NIGMS) and National Cancer Institute (NCI) of the National Institutes of Health (NIH)
 

Remote access: Speed tests

Doing remote data collection and transferring images to your home institution obviously requires fast network connection. But how fast is the 'fast' and is my connection speed sufficient? The tests below may help you to benchmark your connection and compare it against the connection from the University of Michigan that we consider just about appropriate.

NDT tests

The first test we recommend to run is connecting to the ANL Network Diagnostic Tools (NDT). NDT is a web-based server that can be accessed either using a web browser or using a web100clt command-line tool (also known as "ndt-client"). The later can be downloaded from Internet2.

Argonne offers a number of servers to diagnose the effect of the laboratory firewalls:

  http://ndt.anl.gov:7123/   --    behind the ANL firewall, 1Gbps
These NDT speed servers can be used with web100clt if it is installed on your computer:
  web100clt -n prfsnr.aps.anl.gov   --    behind the ANL firewall & the APS divisional firewall, 10Gbps
  web100clt -n 192.5.180.130   --    Residing on the GMCA ScienceDMZ subnet, 10Gbps

Please note that web browser tests may not work with all web browsers and in some case may require installing Java Run Time (JRE) browser plugin. Navigate to one of the above links and start the test. After the test is complete, click on Statistics. Here are typical results for the University of Michigan collected back in 2012. You should expect your speed being not lower than that for successful remote data collection and transfering the images:

WEB100 Enabled Statistics:
Checking for Middleboxes . . . . . . . . . . . . . . . . . .  Done
checking for firewalls . . . . . . . . . . . . . . . . . . .  Done
running 10s outbound test (client-to-server [C2S]) . . . . . 46.09Mb/s
running 10s inbound test (server-to-client [S2C]) . . . . . . 50.80Mb/s

	------  Web100 Detailed Analysis  ------
10 Gbps 10 GigEthernet/OC-192 link found.
Link set to Full Duplex mode
No network congestion discovered.
Good network cable(s) found
Normal duplex operation found.

Web100 reports the Round trip time = 20.06 msec; the Packet size = 1448 Bytes; and
No packet loss was observed.
S2C throughput test: Packet queuing detected: 0.03%
This connection is receiver limited 50.51% of the time.
  Increasing the client's receive buffer (128.0 KB) will improve performance
This connection is sender limited 48.73% of the time.
  Increasing the NDT server's send buffer (127.0 KB) will improve performance
This connection is network limited 0.77% of the time.

If your numbers are significantly lower, that may become a show stopper for remote access. It is important to understand where the problem may occur. Such task is not trivial because in most cases there is no a single bottleneck, but the delays add up non-linearly. In other words, you may have a descent speed between A and B and between B and C, but slow between A and C in spite of the fact that the connection between A and C may be routed via B. Still, it is a good idea to check the speed with some other NDT servers. Here are a few:

List of Internet-2 NDT Servers.
List NDT Servers at the State of Connecticut web site.
CENIC NDT Server.
Stanford University NDT Server.
UC Santa Cruz NDT Server.
Rochester Institute of Technology NDT Server
APS NDT Server (within the APS only; use tests between GMCA & APS).

Traceroute test

In case you cannot perform the NDT tests, they may be substituted by a simpler Traceroute test (on Windows system you need to use tracert.exe). Again, in order to be able to successfully collect data remotely and sft/scp the images to your institution in real time, the times should not be considerably longer than in the test below:

[sstepanov1@watson ~]$ traceroute ndt.anl.gov
traceroute to ndt.anl.gov (146.137.222.101), 30 hops max, 38 byte packets
 1  141.211.182.2 (141.211.182.2)  1.154 ms  1.199 ms  1.221 ms
 2  d-lifsci1-seb.r-seb.umnet.umich.edu (198.108.13.240)  0.666 ms  0.268 ms  0.253 ms
 3  l3-bseb-rseb.r-bin-seb.umnet.umich.edu (192.12.80.131)  0.262 ms  0.244 ms  0.238 ms
 4  v-bin-seb-inet-aa2.merit-aa2.umnet.umich.edu (192.12.80.37)  0.621 ms  0.705 ms  0.533 ms
 5  tenge0-0-0-0x76.nw-chi1.mich.net (198.108.23.13)  7.970 ms  7.804 ms  7.763 ms
 6  192.122.182.18 (192.122.182.18)  18.450 ms  18.527 ms  18.483 ms
 7  sdn-sdsc-anl.anchor.anl.gov (192.5.170.210)  19.425 ms  19.620 ms  19.264 ms
 8  fwzsw-banana-gwv685.anchor.anl.gov (130.202.222.81)  19.311 ms  19.308 ms  19.268 ms
 9  * * *

Note that in some organizations traceroute is blocked by the firewall.

Wget tests

The test below is to compare downloading speeds of the same 88MB file from two different ANL servers. Server-1 resides the ANL public subnet and Server-2 on the GM/CA subnet that is behind an additional APS firewall. The rationale for this test is that we have seen cases when the speed of outside connection to the public ANL servers was fast while the APS firewall was overloaded and then the connection to the GM/CA computers was much slower. Below are typical results for the University of Michigan (you can also use a web browser instead of wget):

wget http://mirror.anl.gov/pub/centos/7/os/x86_64/Packages/libreoffice-core-5.0.6.2-3.el7.x86_64.rpm
........................................................
17:52:44 (5.39 MB/s) ....

wget http://www.gmca.anl.gov/libreoffice-core-5.0.6.2-3.el7.x86_64.rpm
........................................................
16:46:14 (1.49 MB/s) ...

A part of the difference in speeds between mirror.anl.gov and www.gmca.anl.gov is because the later is not a partcularly fast computer, but the APS firewall may have its share in the lower speed too. Therefore, it is important to run the NDT servers with different ANL servers.

SCP tests

Finally, if you already have an SSH access to the GM/CA computers, you may test the real data transfer speeds. Make a directory, e.g. TEST with twenty or so data frames produced by the detector and copy them to your institution. Below are typical speeds for the University of Michigan obtained with 32MB Rayonix-300 CCD frames:

[sstepanov1@watson ~]$ scp -rp sstepanov1@bl3ws5:TEST .
BS_test_0.0001                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0002                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0003                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0004                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0005                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0006                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0007                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0008                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0009                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0010                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0011                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0012                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0013                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0014                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0015                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0016                                   100%   32MB   5.3MB/s   00:06
BS_test_0.0017                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0018                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0019                                   100%   32MB   6.4MB/s   00:05
BS_test_0.0020                                   100%   32MB   5.3MB/s   00:06

To estimate required speeds, one should consider that when data collection is running permanently, the Rayonix frames are typically acquired at the rate of 22fps (frames per minute), which corresponds to 32MB x 22/60 = 11.73MB/s. With Pilatus3-6m and Eiger-16m typical rates are much higher, 50fps (frames per second), which corresponds to 6MB x 50 = 300MB/s and 18MB x 50 = 900MB/s respectively. However, given typical overhead of crystal mounting, centering and screening, average daily data rates are ~200GB/day with Rayonix-300, ~1000GB/day with Pilatus3-6m and ~2000GB/day with Eiger-16m. This amounts to required average speeds of 2MB/s, 10MB/s and 20MB/s with Rayonix, Pilatus and Eiger respectively. In addition, please note that the Eiger detector produces data in the HDF5 format, which are then automatically converted to CBF. If you intent to transfer both formats, expected amount of data and the required speeds for Eiger need to be doubled. Also, with some experiments the amounts of data may be up to several times higher than average. We strongly recommend using Globus for transferring Eiger and Pilatus data.

Improving the speeds

It may be possible to improve network speeds by tuning the TCP parameters on your computer, although the administrative access to your computer and some computer administrating skills may be required. Please read the fasterdata.es.net guide.

 


GM/CA @ APS is an Office of Science User Facility operated for the U.S. Department of Energy Office of Science by Argonne National Laboratory

UChicago Argonne LLC | Privacy & Security Notice | Contact Us | A-Z Index | Search