[bgl-discuss] Information about MPI_MAPPING vs. BGLMPI_MAPPING

[Hari Reddy <hari@xxxxxxxxxx>]

A couple of updates on mapping:

1) MPI_MAPPING is history so please
disregard it.

2) Also, BGLMPI_MAPPING  (for that
matter most of BGLMPI_... ) were never intended for normal users like us.

     You may want to
map mapping to "-mapfile" parameter which is supported by mpirun.

3) There was some confusion about the
default mapping when Virtual Node mode (VN) is used
     to boot the block
(using -mode of mpirun): whether it is TXYZ or XYZT. There were some intentions
     to make this TXYZ,
but that never took place and it looks like this is not going to happen
until next 
     full release. So,
for now it is a good idea to use -mapfile and specify exactly what you
need: TXYZ or XYZT etc
     while operating
under VN mode. 

3) -mapfile 
    Also, this parameter is
a very helpful way to regulate the shape you will be using when
the block size 
    you get allocated happens
to be much larger. For example, if you get allocated a block of size 8X8x8
    and your mesh is:  
4X8X2, then one option is to define the co-ordinates the CPU for each task
so that the all 
    CPUs are contained in
the block of size 4X8X2.
    This information is placed
in a text file and the file name is used as the value for 
    mpirun parameter:  
  -mapfile

    For example, following
the above example, if we have to map 128 tasks to the block of size 4X8X2
which is
    contained in an allocated
block of size: 8X8X8 (booted in VN mode), the file might look like this:

   0 0 0 0
   0 0 0 1
   1 0 0 0
   1 0 0 1
    .......
    ......
    3 7 1 0
    3 7 1 1

The fourth column is used to indicate
cpu 0 (or cpu 1, while operating in VN mode) in a node.

mpirun  -np 128 .... -mapfile <above
text file>

The other options (mentioned above)
for mapfile are:
mpirun -np 128 ... -mapfile TXYZ
or
mpirun -np 128 ..  -mapfile XYZT
 
Hope this helps,

Regards,  Hari

Hari Reddy
ISV Parallel Enablement
IBM High Performance Computing Solutions Development
IBM Systems and Technology Group
Tel:  817-962-6472 (T/L 522-6472)
hari@xxxxxxxxxx

"Dinesh Kaushik" <kaushik@xxxxxxxxxxx>
Sent by: owner-discuss@xxxxxxxxxxxxxxx

08/29/2005 05:39 PM

To	"Narayan Desai" <desai@xxxxxxxxxxx>, "Susan Coghlan" <smc@xxxxxxxxxxx>
cc	<discuss@xxxxxxxxxxxxxxx>, <support@xxxxxxxxxxxxxxx>
Subject	Re: [bgl-discuss] Re: [bgl-support] MPI_MAPPING vs. BGLMPI_MAPPING

Narayan,

With the new version of cobalt, the mapping option (-e BGLMPI_MAPPING=TXYZ)
works as 
expected now. The ouput shows the following for the two mapping env variables:

BGLMPI_MAPPING = TXYZ, MPI_MAPPING = NONE

Thanks for your effort on fixing this problem.

Dinesh

----- Original Message ----- 
From: "Dinesh Kaushik" <kaushik@xxxxxxxxxxx>
To: "Narayan Desai" <desai@xxxxxxxxxxx>; "Susan Coghlan"
<smc@xxxxxxxxxxx>
Cc: <discuss@xxxxxxxxxxxxxxx>; <support@xxxxxxxxxxxxxxx>
Sent: Monday, August 01, 2005 10:35 PM
Subject: Re: [bgl-discuss] Re: [bgl-support] MPI_MAPPING vs. BGLMPI_MAPPING


> Narayan and Susan,
>
> After testing on a dedicated partition, I conclude that the current
mpirun is 
> broken. It generates an error if BGLMPI_MAPPING is specified:
>
> <Aug  1 22:24:07>  FE_MPI (ERROR): You have used both
BGLMPI_MAPPING and 
> MPI_MAPPING env. vars
>
> Specifying MPI_MAPPING=TXYZ does not change the default mapping (XYZT).
>
> Right now, the only way to change the mapping seems through -mapfile
option for 
> mpirun. This works as expected. However, we do not have this option
supported 
> through the scheduler (please correct me if I am wrong).
>
> Thanks,
>
> Dinesh
>
> ----- Original Message ----- 
> From: "Narayan Desai" <desai@xxxxxxxxxxx>
> To: "Katherine M Riley" <kmriley@xxxxxxxxxxxxxxxxxx>
> Cc: <discuss@xxxxxxxxxxxxxxx>; <support@xxxxxxxxxxxxxxx>
> Sent: Thursday, July 28, 2005 3:28 PM
> Subject: [bgl-discuss] Re: [bgl-support] MPI_MAPPING vs. BGLMPI_MAPPING
>
>
>>>>>>> "Katherine" == Katherine M Riley
<Katherine> writes:
>>
>>  Katherine> For the tutorial we were trying to demonstrate
some
>>  Katherine> mapping options.  But, when folks were
passing in with
>>  Katherine> "-e" to the queuing system, we would
get a complaint that
>>  Katherine> both MPI_MAPPING and BGLMPI_MAPPING are both
set.  None
>>  Katherine> of us know what MPI_MAPPING is but we tried
setting that
>>  Katherine> back and forth between XYZT and TXYZ and got
no change.
>>
>>  Katherine> So - my question - do we know what MPI_MAPPING
is?  And,
>>  Katherine> why can't we set BGLMPI_MAPPING?
>>
>> Cobalt isn't setting any environment variables implicitly. My
hunch is
>> that this is a bug in the mpirun command. We could concievably
remap
>> this option in the process manager, if we can find a working way
to
>> setup the mapping.
>>
>> Has the environment variable name changed? Does specifying the
>> MPI_MAPPING variable work?
>> -nld
>>
>>
>>
>> - --------------------------------------------------------------------
>> To add or remove yourself from this mailing list, use the 'notifyme'
>> command on any BGL machine. To remove: notifyme -n, to add: notifyme
-y.
>>
>>
>
> - --------------------------------------------------------------------
> To add or remove yourself from this mailing list, use the 'notifyme'
> command on any BGL machine. To remove: notifyme -n, to add: notifyme
-y.
>
> 

- --------------------------------------------------------------------
To add or remove yourself from this mailing list, use the 'notifyme'
command on any BGL machine. To remove: notifyme -n, to add: notifyme -y.