[bgl-discuss] discrete MPI_Wtime offset

[Anthony Chan <chan@xxxxxxxxxxx>]

I notice that the MPI_Wtime returned by BGL-MPI has a ~0.11 msec discrete
offset for every 32 processes (BGL's MPI_WTIME_IS_GLOBAL is true).  This
can be observed in MPI job with number of nodes, np = 64 and 256 but
_NOT_ in np = 128 (strange?).   For np=256, the maximum time offset
~0.78 msec.  The output of the following simple MPI program:

>int main( int argc, char *argv[] )
>{
>
>     int          world_size, world_rank;
>     double       chk_time_zero, chk_time_start;
>
>     MPI_Init( &argc, &argv );
>
>     chk_time_zero  = MPI_Wtime();
>
>     MPI_Barrier( MPI_COMM_WORLD );
>     MPI_Barrier( MPI_COMM_WORLD );
>
>     /* 2nd MPI_Wtime after 2 barriers should be approx. aligned */
>     chk_time_start  = MPI_Wtime();
>
>     MPI_Comm_rank( MPI_COMM_WORLD, &world_rank );
>     MPI_Comm_size( MPI_COMM_WORLD, &world_size );
>
>     printf( "%d : time0 = %lf, time1 = %lf\n",
>             world_rank, chk_time_zero, chk_time_start );
>
>     MPI_Finalize();
>     return 0;
>}

illustrates the jumps.  Here is time sorted output of a 256-node run of
the above program compiled by mpxlc-7.0:

13 : time0 = 50.567090, time1 = 50.567112
...
54 : time0 = 50.567095, time1 = 50.567112
253 : time0 = 50.567201, time1 = 50.567224
...
247 : time0 = 50.567208, time1 = 50.567224
128 : time0 = 50.567316, time1 = 50.567334
...
186 : time0 = 50.567321, time1 = 50.567334
193 : time0 = 50.567428, time1 = 50.567446
...
250 : time0 = 50.567433, time1 = 50.567446
0 : time0 = 50.567537, time1 = 50.567555
...
58 : time0 = 50.567542, time1 = 50.567556
64 : time0 = 50.567649, time1 = 50.567667
...
122 : time0 = 50.567654, time1 = 50.567667
133 : time0 = 50.567758, time1 = 50.567779
...
182 : time0 = 50.567763, time1 = 50.567779
124 : time0 = 50.567869, time1 = 50.567890
...
119 : time0 = 50.567873, time1 = 50.567889

When viewing the slog2 file of the above program in Jumpshot, one will
notice that MPI_Barrier states are not aligned.  For program with many
send/recv of small messages, there will be backward arrows in Jumpshot.
My guess is that this is a bug in BGL's MPI implementation.  Any comment
is welcome.


A.Chan

- --------------------------------------------------------------------
To add or remove yourself from this mailing list, use the 'notifyme'
command on any BGL machine. To remove: notifyme -n, to add: notifyme -y.