We have a number of supercomputers connected to the CASA network. These include the Cray C90 and Intel Paragon at SDSC, the Intel Delta and Paragon at the California Institute of Technology (Caltech), and a Cray Y-MP at Jet Propulsion Laboratories (JPL). The Cray Y-MP and C90 are capable of driving a HIPPI channel at or close to full bandwidth, using full sized IP packets (64 KB) because these machines are limited by per-packet interrupt latencies.
We have reported previously on TCP performance tests performed on the SDSC-Caltech CASA link, between 2 Cray supercomputers. The delay-bandwidth product (DBP) of the SDSC-Caltech link is approximately 0.3 MB, or about four packets of 64 KB each in flight. The DBP product on the SDSC - LANL link is much higher, given the longer geographical distance and hence latency. We observe a 30 msec. RTT for 64 KB IP packets, giving us a DBP of approximately 2 MB. We use the TCP window scaling option for TCP, allowing the Crays to use the entire available TCP bandwidth if possible. We additionally had to increase the Cray advertised TCP window beyond 378 KB (which is the maximum available in default configuration) by increasing the amount of memory allocatable to socket mbufs.
With the following specifications,
We achieved 500 Mbps TCP sustained throughput (as reported by TTCP and netperf). Note that the test occurred in production mode on the supercomputers, i.e., CPU resources were timeshared amongst competing applications. Large supercomputers such as the SDSC Cray are timeshared machines, with a CPU scheduler controlling the sharing of CPU resources. The effect of scheduling on a network-intensive job such as a large bulk transfer can be critical; interrupting such jobs adds significant delay to packet transmissions and degrades performance.