[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: [bgl-discuss] Can BGL jobs wind up working with corrupted memory?
BG/L dumped a core file when I ran a program that causes the stack overflow.
it seems that the system dumps only when an user program violates
the OS working area (not for user program's heap area).
>>Yes, the BG/L software should automatically report a clear error,
>>explaining where the application ran out of memory, and what the call
>>stack looked like.
>>
>>
>
>I agree that the system should provide protection mechanism.
>
>
>
>>Everything else is just a work-around for an obvious IBM bug.
>>
>>
>
>Esp. for computational program, there are stack consuming things
>such as recursive function.
>
>I executed a program like chkstack.c (in attachment).
>It just exited with "Job terminated normally",
>instead of "Stack Overflow". This is completely "abnormal exit".
>
>[0]: stack usage = 533594112 bytes
>[0]: stack usage = 533725184 bytes
><May 20 08:53:08> FE_MPI (Info) : Job terminated normally
>
>* 533725184 bytes -> 509MB
>
>
>BTW, chkstack just causes seg.v with a msg like "stack overflow"
>on a regular system. On my linux box, it hanged at around 10M stack usage.
>
>
>
>>That said, the TAU memory tool for measuring "memory headroom" during
>>an application is very nice. It works with Fortran, and is more than
>>just a tool to address this particular IBM bug. It is really a tool
>>for understanding how your memory gets used during your program
>>execution, and where you reach memory "low water marks".
>>
>>For example, you may find out that one routine is consistently a
>>memory hog, and that fixing that area could let you crank the problem
>>size for all the other parts of the code.
>>
>>With these small-memory machines, the problem will be around for a
>>while. Machines built on the cell processor will also require a lot of
>>memory management from both the compiler, run-time system, and
>>middleware.
>>
>>-Pete
>>
>>At 4:34 PM -0500 5/19/05, Steven Pieper wrote:
>>
>>
>>
>>>What really bothers me is that this `feature' of the BGL kernal seems
>>>to allow for incorrect numerical results with no warning. The user
>>>has to be aware that this problem might be happening and build in
>>>checks, if that is possible. How do I know how big to increase
>>>the program's data space? Ho do I know how much of the stack
>>>or heap the compiler is using in each subroutine? etc.
>>>
>>>steve
>>>
>>>
>>>
>>>
>>>
>>>>>>Hi,
>>>>>>
>>>>>>problem is that BG/L compute node kernel doesn't check stack overflow
>>>>>>However, I believe that the OS can check overflow of heap memory.
>>>>>>In fact,of course, the malloc() routine on BG/L returns 0 if run
>>>>>>out of
>>>>>>memory.
>>>>>>
>>>>>>Fortunatelly, sbrk() is available on BG/L glibc.
>>>>>>so we can increment the program data space to
>>>>>>reduce possiblity of stack overflow which eats out of the heap
>>>>>>memory.
>>>>>>
>>>>>>For example, in C code,
>>>>>>
>>>>>>sbrk( 1024*1024 ); // increments the program's data space by 1M bytes
>>>>>>
>>>>>>Sorry I don't know how to call glibc function such as sbrk() from
>>>>>>Fortran.
>>>>>>
>>>>>>
>>>>>>I wrote a small C code to make sure sbrk() works or not.
>>>>>>
>>>>>>This is not perfect solution but it works.
>>>>>>The perfect solution is to add stack overflow check to
>>>>>>BG/L compute node kernel.
>>>>>>
>>>>>>
>>>>>>Sorry if I miss the point..
>>>>>>
>>>>>>
>>>>>>------------------------
>>>>>>#include <stdio.h>
>>>>>>#include <stdlib.h>
>>>>>>#include <unistd.h>
>>>>>>
>>>>>>int main(int argc, char* argv[])
>>>>>>{
>>>>>>unsigned long size,totalsize,s;
>>>>>>void *p;
>>>>>>printf("sbrk()=%p\n", sbrk(0));
>>>>>>if( argc > 1 ) {
>>>>>>int inc_kb;
>>>>>>inc_kb = atoi(argv[1]);
>>>>>>printf("inc. stack size %dkb\n", inc_kb);
>>>>>>sbrk(inc_kb*1024);
>>>>>>printf("sbrk()=%p\n", sbrk(0));
>>>>>>}
>>>>>>size = 256*1024*1024;
>>>>>>totalsize = 0;
>>>>>>
>>>>>>for(s=size ; s>1024; s>>=2 ) {
>>>>>>for(;;) {
>>>>>>p = malloc( s );
>>>>>>if( !p ) break;
>>>>>>totalsize += s;
>>>>>>}
>>>>>>}
>>>>>>printf("%6.2f MB alloc'ed\n", (double)totalsize/1024.0/1024.0);
>>>>>>return 0;
>>>>>>}
>>>>>>
>>>>>>
>>>>>># bgl/BlueLight/ppcfloor/blrts-gnu/bin/powerpc-bgl-blrts-gnu-gcc
>>>>>>-Wall a.c
>>>>>># cqsub -t 10 -n 1 a.out
>>>>>># cat 2958.output
>>>>>>[0]: sbrk()=0x162740
>>>>>>[0]: inc. stack size 1024kb
>>>>>>[0]: sbrk()=0x278000
>>>>>>[0]: 507.94 MB alloc'ed
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>>Does this help me with my Fortran application?
>>>>>>>
>>>>>>>Even if it does, is IBM being complained to? I find it completely
>>>>>>>
>>>>>>>
>>>>>>unacceptable
>>>>>>
>>>>>>
>>>>>>>if the user has to monitor that the operating system is
>>>>>>>
>>>>>>>
>>>>>>mismanaging memory.
>>>>>>
>>>>>>
>>>>>>>Thanks
>>>>>>>Steve
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>>>Hi Pete,
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>Last week the TAU folks visited, and we discussed a new
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>memory tool
>>>>>>
>>>>>>
>>>>>>>>>>>which could give you a "headroom" measurement over the life
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>of your
>>>>>>
>>>>>>
>>>>>>>>>>>program, essentially looking at the difference between the
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>stack and
>>>>>>
>>>>>>
>>>>>>>>>>>heap and how much you have left. You could then see which
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>routines
>>>>>>
>>>>>>
>>>>>>>>>>>put you closest to overflow.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>Such a tool is in the works....
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>-Pete
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>We're released TAU v2.14.4 with support for memory headroom
>>>>>>>>>>
>>>>>>>>>>
>>>>>>evaluation options.
>>>>>>
>>>>>>
>>>>>>>>>>I've enclosed a description of the various options below.
>>>>>>>>>>http://www.cs.uoregon.edu/research/paracomp/tau
>>>>>>>>>>Thanks,
>>>>>>>>>>- Sameer
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>-
>>>>>>>
>>>>>>>
>>>>>>--------------------------------------------------------------------
>>>>>>
>>>>>>
>>>>>>>To add or remove yourself from this mailing list, use the 'notifyme'
>>>>>>>command on any BGL machine. To remove: notifyme -n, to add:
>>>>>>>
>>>>>>>
>>>>>>notifyme -y.
>>>>>>
>>>>>>
>>>>>>-
>>>>>>--------------------------------------------------------------------
>>>>>>To add or remove yourself from this mailing list, use the 'notifyme'
>>>>>>command on any BGL machine. To remove: notifyme -n, to add:
>>>>>>notifyme -y.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>- --------------------------------------------------------------------
>>>To add or remove yourself from this mailing list, use the 'notifyme'
>>>command on any BGL machine. To remove: notifyme -n, to add: notifyme -y.
>>>
>>>
>>
>>
>>
>
>
>
>------------------------------------------------------------------------
>
>#include <stdio.h>
>
>struct stupid {
> char buf[1024*128];
>};
>
>static unsigned long stack_usage=0;
>
>static void foo( struct stupid a )
>{
> stack_usage += sizeof( struct stupid );
> fprintf(stderr, "stack usage = %lu bytes\n", stack_usage );
> foo( a );
>}
>
>int main(int argc, char *argv[])
>{
> struct stupid a;
> fprintf(stderr, "sizeof stupid=%d\n", sizeof(a) );
> foo(a);
> return 0;
>}
>
>
software signal..................0x0000000b (SIGSEGV - segmentation violation)
generated by interrupt...........0x00000002 (data storage interrupt)
while executing instruction at...0x00106c64
dereferencing memory at..........0x00000046
general purpose registers:
r00=0x00100bf8 r01=0x0012a5c0 r02=0x1eeeeeee r03=0x00000000
r04=0x0013e72c r05=0x0012ac30 r06=0x00148318 r07=0x0000001f
r08=0xffffffe0 r09=0x00160000 r10=0x00000000 r11=0x00160000
r12=0x28000888 r13=0x1eeeeeee r14=0xffffffff r15=0xffffffff
r16=0x00000000 r17=0x00089b10 r18=0x00089b08 r19=0x000887ec
r20=0xfffffff4 r21=0x00000008 r22=0x00040000 r23=0x00000000
r24=0x00000000 r25=0x001000a8 r26=0x0012ac30 r27=0x00000000
r28=0x0013e72c r29=0x0014ac70 r30=0x00089ae0 r31=0x0012a5c0
special purpose registers:
lr=0x00100bf8 cr=0x28000888 xer=0x20000002 ctr=0x00105d08
memory:
stack top............0x1fec0000
stack frame pointer..0x0012a5c0
end of heap..........0x00162640
start of program.....0x00100000
Personality
XYZ Coordinates..... 0, 0, 0
MPI Rank............ 0
# of interrupts... 4074
Interrupt History (current TB=00000003 532767fd)
TB=0000000349a0357f iar=00114a24 sp=00208560 (system call interrupt)
TB=0000000349a295d7 iar=00114a24 sp=001e8540 (system call interrupt)
TB=0000000349a4f81f iar=00114a24 sp=001c8520 (system call interrupt)
TB=0000000349a75ae3 iar=00114a24 sp=001a8500 (system call interrupt)
TB=0000000349a9bc63 iar=00114a24 sp=001884e0 (system call interrupt)
TB=0000000349ac1c83 iar=00114a24 sp=001684c0 (system call interrupt)
TB=0000000349ae7eb7 iar=00114a24 sp=001484a0 (system call interrupt)
TB=0000000349b0d92b iar=00106c64 sp=0012a5c0 (data storage interrupt)
DCRs:
DCR BGL_TSDCR_RE_SND_XP = 0
DCR BGL_TSDCR_RE_SND_XM = 0
DCR BGL_TSDCR_RE_SND_YP = 0
DCR BGL_TSDCR_RE_SND_YM = 0
DCR BGL_TSDCR_RE_SND_ZP = 0
DCR BGL_TSDCR_RE_SND_ZM = 0
Function call chain:
0x00106c64
0x2c02fffc
0x00100bf4
0x00100564
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
0x00100590
End of stack