[bgl-discuss] BGL compute-to-login socket performance

[Rob Ross <rross@xxxxxxxxxxx>]

Hi all,

I hacked up NetPIPE (http://www.scl.ameslab.gov/netpipe/) to work on the
BG/L compute nodes so that I could see what sort of bandwidth we could get
from a compute out to a login node through the system call redirection
system.  I used the open() trick mentioned before.

Here's a graph:

  http://www.mcs.anl.gov/~rross/bgl-socket-netpipe.png

We hit a maximum of around 400Mbit/sec starting around 512KB messages, and 
we get half of that around 10KB messages.

Given that we're traversing both the tree (~1Gbit/sec IIRC) and the GigE 
networks in this test, I'm not all that disappointed with the results.

One of the BG/L developers seemed surprised that I wasn't able to get 
connect() to work, so perhaps I'm just not doing things quite right on 
that front.  In any case, this is the performance I would expect to see 
from a "real" socket as well.

Because the network is similar, I would expect performance from compute 
node to file system node to be roughly the same as well.

As an aside, it also passes NetPIPE's "integrity check".

For those of you that might be interested, this is a drop-in replacement 
for the "src/tcp.c" in the NetPIPE distribution:

  http://www.mcs.anl.gov/~rross/tcp-hacked-for-bgl.c

I simply compiled with mpicc for the compute node, and ran the default tcp 
version on the login node compiled with gcc.

Rob

- --------------------------------------------------------------------
To add or remove yourself from this mailing list, use the 'notifyme'
command on any BGL machine. To remove: notifyme -n, to add: notifyme -y.