Re: [bgl-discuss] global tree bandwidth and socket write() performance

[Kamil Iskra <iskra@xxxxxxxxxxx>]

On Fri, Sep 23, 2005 at 23:04:23 -0500, Chad Glendenin wrote:

> When I write() chunks of the buffer myself, I get a little over 60 MB/sec
> from a single compute node to a front-end node, but when I let the OS
> handle it, I only get about 2.8 MB/sec.
[snip]
> When I let the OS do it, it writes either 72400 or 73848 bytes at a time,
> and it's over 20 times slower, even though I'm calling write() more often
> when I use a payload less than 72400 bytes. (I also tried payloads of 80
> KB.)

I have a theory why that might be the case:

As you undoubtedly know, there is no full-blown OS running on the compute
nodes.  Therefore, all your write() calls are forwarded to the I/O node and
executed there.

What might be happening, is that every time you call write() with a large
buffer, the whole buffer gets sent to the I/O node.  Since, as you found
out, write() only writes some 72K of it, the remaining part of the buffer
that has been sent would be unused.  So you call write() again, with a
large buffer again, and that data gets resent.  This repeated sending of
the write() buffer could account for the performance drop you observe.

There is a way to verify it.  ZeptoOS 1.1 (installed on ANL BG/L) lets you
log on the I/O node and strace the "ciod" process (you will need root
access to be able to do the latter).  You would then find out how large a
buffer is being passed to the write() calls.  I could check it myself, but
I don't have a suitable benchmark at hand, and I am too lazy to write one
right now (it is Monday morning after all.. :-).

Let us know what you find out, or if you need any help.

Greetings,

Kamil

-- 
Kamil Iskra, PhD
Argonne National Laboratory, Division of Mathematics and Computer Science
9700 South Cass Avenue, Argonne, IL 60439, USA
phone: +1-630-252-7197  fax: +1-630-252-5986

- --------------------------------------------------------------------
To add or remove yourself from this mailing list, use the 'notifyme'
command on any BGL machine. To remove: notifyme -n, to add: notifyme -y.