[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: [bgl-discuss] netcdf compilation on BGL



The Makefile for Dacapo is attached with this email and the output of
the make -n command is as follows -

anilekar@login1:~/Installs/trials/6/Dacapo-2.7.7/src>gmake -n
echo    'Usage: gmake [arch] [ MP=MP_library ]'
echo    'where arch is one of'
echo
echo    'generic                (should compile on all machines, '
echo   '                        see notes on FFTW and blas)'
echo    'sun_ultra              (Sun ultra-1 or ultra-2)'
echo    'sun_ss10               (Sun sparcstation 10/20)'
echo    'ibm_power3             (RS/6000 power3 node)'
echo    'ibm_power2_wide        (RS/6000 wide power2 node: 590,591,...)'
echo    'ibm_power2_thin        (RS/6000 thin/thin2 power2 node:
390,...)'
echo    'ibm_power              (Older RS/6000 power cpus: 580,...)'
echo    'ibm_ppc                (RS/6000 powerPC)'
echo    'vpp500                 (Fujitsu VPP-500)'
echo   'cray                   (CRAY UNICOS)'
echo   'sgi                    (Silicon Graphics n32 ABI)'
echo   'alpha                  (Digital Alpha)'
echo   'pglinux                (Portland Group pgf90 compiler on Linux)'
echo   'intellinux             (Intel ifc Fortran compiler version >=6.0
on Linux)'
echo
echo    'and the supported MP_libraries so far are serial and mpi'
echo    '(default is MP_library=serial).'
 

I use the ibm_power3 option for my installation.

Thanks,
Anand

-----Original Message-----
From: Andrew Siegel [mailto:asiegel@xxxxxxxxxxxxxxx] 
Sent: Tuesday, July 12, 2005 3:03 PM
To: Anand Nilekar
Cc: discuss@xxxxxxxxxxxxxxx
Subject: RE: [bgl-discuss] netcdf compilation on BGL


can you show me your Makefile. Or, better yet, the output of make -n
(after a make clean). I want to elminate the obvious things first.

On Tue, 12 Jul 2005, Anand Nilekar wrote:

> Hi Andrew,
>
> Here is the error msg I get which contains the list of symbols that
are
> not found in the ~walkup version --
>
> ../netcdf/ibm_power3/libnetcdf.a(ncio.o)(.text+0x1126): In function
> `px_pgin':
> : undefined reference to `errno'
> ../netcdf/ibm_power3/libnetcdf.a(ncio.o)(.text+0x113e): In function
> `px_pgin':
> : undefined reference to `errno'
> ../netcdf/ibm_power3/libnetcdf.a(ncio.o)(.text+0x114a): In function
> `px_pgin':
> : undefined reference to `errno'
> ../netcdf/ibm_power3/libnetcdf.a(ncio.o)(.text+0x116e): In function
> `px_pgin':
> : undefined reference to `errno'
> ../netcdf/ibm_power3/libnetcdf.a(ncio.o)(.text+0x11be): In function
> `px_pgin':
> : undefined reference to `errno'
> ../netcdf/ibm_power3/libnetcdf.a(ncio.o)(.text+0x129e): more undefined
> references to `errno' follow
> ../netcdf/ibm_power3/libnetcdf.a(string.o)(.text+0xe): In function
> `toupper':
> : undefined reference to `__ctype_toupper'
> ../netcdf/ibm_power3/libnetcdf.a(string.o)(.text+0x1a): In function
> `toupper':
> : undefined reference to `__ctype_toupper'
> ../netcdf/ibm_power3/libnetcdf.a(string.o)(.text+0x32): In function
> `tolower':
> : undefined reference to `__ctype_tolower'
> ../netcdf/ibm_power3/libnetcdf.a(string.o)(.text+0x3e): In function
> `tolower':
> : undefined reference to `__ctype_tolower'
> ../netcdf/ibm_power3/libnetcdf.a(string.o)(.text+0x4f6): In function
> `NC_check_name':
> : undefined reference to `__ctype_b'
> ../netcdf/ibm_power3/libnetcdf.a(string.o)(.text+0x502): In function
> `NC_check_name':
> : undefined reference to `__ctype_b'
>
>
>
>
> Thanks,
> Anand
>
>
>
> -----Original Message-----
> From: Andrew Siegel [mailto:asiegel@xxxxxxxxxxxxxxx]
> Sent: Tuesday, July 12, 2005 2:43 PM
> To: Anand Nilekar
> Cc: discuss@xxxxxxxxxxxxxxx
> Subject: Re: [bgl-discuss] netcdf compilation on BGL
>
>
> Can you please list the symbols that cannot be located when you try to
> link to the ~walkup version?
>
>
> On Tue, 12 Jul 2005, Anand Nilekar wrote:
>
> > Hi,
> >
> > I have been trying to install my code (Dacapo -
> > http://dcwww.camp.dtu.dk/campos//Dacapo/) on BGL. The code
> successfully
> > compiles till the point it starts linking libraries. Previously I
had
> > problems with ESSL and MASS math libraries. But now I am using blas
> > instead (as Katherine suggested), which doesn't seem to give me any
> > problem. However, the netcdf installation I have lacks some required
> > functions (fortran related). I also tried the library in
> > /home/walkup/local/netcdf-3.5.1/lib/libnetcdf.a as I found it on the
> ANL
> > wiki page. But that also lacks some functions, although different
from
> > what my installation doesn't have.
> >
> > When I looked at my netcdf install log closely, I found that it
can't
> > figure out the fortran related variables, e.g.
> >
> > >>checking "xlf" as Fortran-90 compiler... failed to build test
> program.
> >
> > I tried various variables for FC and FFLAGS, but couldn't get it to
> > work. The rest of the env vars that work are defined as follow,
which
> I
> > hope are correct as I didn't see any problem with these.
> >
> > setenv CPPFLAGS '-D_HPUX_SOURCE -D_FILE_OFFSET_BITS=64'    # large
> file
> > support
> > setenv CFLAGS '-g +DD64'                           # 64-bit mode
> > setenv FLIBS '-lU77'
> > setenv CXX ''                                      # no 64-bit mode
> C++
> > compiler
> >
> >
> >
> > Does anyone know what is missing here? Or have encountered similar
> > problems before? Any pointers are highly appreciated.
> >
> >
> >
> > Thanks in advance,
> >
> > Anand
> >
> >
> >
> >
> > -
--------------------------------------------------------------------
> > To add or remove yourself from this mailing list, use the 'notifyme'
> > command on any BGL machine. To remove: notifyme -n, to add: notifyme
> -y.
> >
> >
>
>
>
# This Makefile works together with GNU make.
# GNU make can be obtained from any FTP archive near you.
# The Danish FTP archive currently has the GNU make in the file:
#    ftp://ftp.denet.dk/pub/gnu/make-3.75.tar.gz

# The concept used is to compile for each "arch" in a separate
# subdirectory OBJECTDIR (arch_serial,arch_mpi).
# Either serial or message-passing versions may be compiled.

default:
        @echo   'Usage: gmake [arch] [ MP=MP_library ]'
        @echo   'where arch is one of'
        @echo
        @echo   'generic                (should compile on all machines, '
        @echo   '                        see notes on FFTW and blas)'
        @echo   'sun_ultra              (Sun ultra-1 or ultra-2)'
        @echo   'sun_ss10               (Sun sparcstation 10/20)'
        @echo   'ibm_power3             (RS/6000 power3 node)'
        @echo   'ibm_power2_wide        (RS/6000 wide power2 node: 590,591,...)'
        @echo   'ibm_power2_thin        (RS/6000 thin/thin2 power2 node: 390,...)'
        @echo   'ibm_power              (Older RS/6000 power cpus: 580,...)'
        @echo   'ibm_ppc                (RS/6000 powerPC)'
        @echo   'vpp500                 (Fujitsu VPP-500)'
        @echo   'cray                   (CRAY UNICOS)'
        @echo   'sgi                    (Silicon Graphics n32 ABI)'
        @echo   'alpha                  (Digital Alpha)'
        @echo   'pglinux                (Portland Group pgf90 compiler on Linux)'
        @echo   'intellinux             (Intel ifc Fortran compiler version >=6.0 on Linux)'
        @echo
        @echo   'and the supported MP_libraries so far are serial and mpi'
        @echo   '(default is MP_library=serial).'

#####################################################################
#
# Common definitions
#

# Name of this Makefile:
MAKEFILE = Makefile

# Message-passing library type.  Default is "none", i.e., MP=serial:
MP = serial

# Should swap-file be used for the qrad array (should be set for this version)
QRAD_SWAP_OPT = -DQRAD_SWAP

# Source directory is one level above:
SRCDIR = ..

# Directory for libraries and .o files
OBJECTDIR = ${ARCH}_${MP}

COMMONBINS =  generate_kpoints.o  read_structure_netcdf.o bastr.o  chdop.o  efertr.o  mach.o \
       fexcop.o genbop.o nlproj.o orthop.o wfinop.o density_mixing.o \
       setuop.o ionmov.o ion_conjgrad.o gen_coord.o auxmod.o potion.o fft3d2.o \
       forloc.o calvps.o inidir.o eferop.o eisdia.o \
       dipole.o kptana.o resmin.o \
       corec.o pulay.o davidson.o qrutil.o \
       gen_orthop.o \
       qvan2.o setuspot.o us_calcqr.o \
       us_hpsi.o us_newd.o ylmr2.o gk_sort.o \
       rmm_diss.o wfcinit.o potinit.o us_force.o erfc.o \
       eigsolve.o subproj.o bfgs.o \
       tmp_read.o matrix_utilities.o \
       fftdimensions.o ewald_module.o dacapo.o update_netcdf_history_module.o \
       double_grid.o real_nonlocal.o applyh.o angula.o stress.o madelung.o \
       blochlmodule.o external_pot_module.o  addusdens.o pbe.o pbeunsp.o \
       pw.o pwunsp.o  symana.o mulcen.o gen_hamiltonian_matrix.o

SERIALBINS =  run_context.o van_us_data_module.o ${COMMONBINS} damden.o libtos.o  

PARALLELBINS = run_context.o van_us_data_module.o par_functions.o ${COMMONBINS} damden.o libtos.o  ms.o mpi.o par_getwf.o 

# Make-directory command
MKDIR = mkdir -p

# Netcdf include dir 
NETCDFINC = -I../netcdf/${ARCH} -I../netcdf

# If a efficient FFT library is not available (like ESSL on IBM RS/6000 
# or DXML on Digital Alpha) dacapo can be linked together with 
# FFTW Fast Fourier Transform library. 
# Download and compile the FTTW library (version 2.1.3) from www.fftw.org 
# (remember to add -lfftw to LIBS_arch) 
USEFFTW=-DFFTW

#####################################################################
#
# Machine-specific rules
#
# ---------------------- Generic  ---------------------
# To compiled using 'gmake generic' requires a version
# of the BLAS library and version 2.1.3 of the FFTW library.
# In the GENERIC section of mach.F a time function can be added,
# and the flush function can be changed.
# The library for netcdf libnetcdf.a must be copied to
#     src/netcdf/include/generic/libnetcdf.a

GENERIC_DEFINES =  ${USEFFTW} ${QRAD_SWAP_OPT} -DGENERIC
GENERIC_OPT = -fast 
GENERIC_FFLAGS = -byteswapio -Msecond_underscore -g77libs
LIBS_generic=-lblas -lfftw
LIBS_generic_mpi= -L/usr/share/mpi/include/  -lmpich
GENERIC_MPIDIR=/home/data/lhansen/MPICH/linux/mpich-1.2.1
 
generic:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=generic\
        FC=pgf90 FC90=pgf90 F90FF='-M free ' MPIFC='${GENERIC_MPIDIR}/bin/mpif90 -f90=pgf90' MPIDIR=${GENERIC_MPIDIR} \
        FFLAGS="${GENERIC_OPT} ${GENERIC_FFLAGS} ${GENERIC_DEFINES}" \
        DEFINES="${GENERIC_DEFINES}" \
        LIBS= ${LIBS_generic} \
        INCLUDES='-I${SRCDIR} -I../netcdf/generic -I../netcdf' \
        EXTRABINS='fargc.o'                                                       

# ---------------------- pglinux : Portland Group pgf90 running on Linux -------------
PGLINUX_DEFINES =  ${USEFFTW} ${QRAD_SWAP_OPT} -DPGLINUX

PGLINUX_OPT = -fast

# PGLINUX_FFLAGS = -byteswapio -Msecond_underscore -g77libs -tp=p7
PGLINUX_FFLAGS = -byteswapio -Msecond_underscore -Bstatic -tp=p7

# using Atlas blas
LIBS_pglinux=-llapack -lcblas -lf77blas -latlas /home/niflheim/lhansen/fftw/install/lib/libfftw.a

# using MKL
# MKLPATH="/opt/intel/mkl70/lib/32/"
# LIBS_pglinux=/home/niflheim/lhansen/fftw/install/lib/libfftw.a ${MKLPATH}/libmkl_lapack.a ${MKLPATH}/libmkl_ia32.a ${MKLPATH}/libg
uide.a -lpthread

# using LAM-MPI
# LIBS_pglinux_mpi= -L/usr/local/lib/  -llamf77mpi
# LIBS_pglinux_mpi= -L/usr/local/lam-6.5.9-pgi/lib  -llamf77mpi
# PGLINUX_MPIDIR=/usr/local/lam-6.5.9-pgi/
LIBS_pglinux_mpi= -L/home/camp/lhansen/thul/pgi/lam/install/lib  -llamf77mpi
PGLINUX_MPIDIR=/home/camp/lhansen/thul/pgi/lam/install/

 
pglinux:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=pglinux\
        FC=pgf90 FC90=pgf90 F90FF='-Mfree ' MPIFC='${PGLINUX_MPIDIR}/bin/mpif77 ' MPIDIR=${PGLINUX_MPIDIR} \
        FFLAGS="${PGLINUX_OPT} ${PGLINUX_FFLAGS} ${PGLINUX_DEFINES}" \
        DEFINES="${PGLINUX_DEFINES}" \
        LIBS= ${LIBS_pglinux} \
        INCLUDES='-I${SRCDIR} -I../netcdf/pglinux -I../netcdf' \
        EXTRABINS='fargc.o'                                                                

# ---------------------- intellinux : Intel ifc running on Linux -------------
# Information at http://developer.intel.com/software/products/eval/
# Intel support at http://support.intel.com/support/performancetools/fortran/v6/Linux/

INTELLINUX_DEFINES =  ${USEFFTW} ${QRAD_SWAP_OPT} -DINTELLINUX
# INTELLINUX_OPT = 
# For highest optimization on Pentium-4 (see "ifc -help"):
INTELLINUX_OPT = -O3 -tpp7 -xW
# INTELLINUX_OPT = -C -g
INTELLINUX_FFLAGS = -cm -w
# LIBS_intellinux=-lblas -lfftw
# LIBS_intellinux=-llapack -lcblas -lf77blas -latlas -lfftw -lPEPCF90 
LIBS_intellinux=-lcblas -lf77blas -latlas -lfftw -lPEPCF90 
LIBS_intellinux_mpi= -L/usr/local/mpi  -lmpich
INTELLINUX_MPIDIR=/usr/local/mpi
 
intellinux:
        @ echo Creating the intellinux_${MP}/work.pcl file for the ifc compiler
        @if [ ! -d intellinux_${MP} ]; then \
            ${MKDIR} intellinux_${MP}; \
        fi
        @ echo 'work.pc' > intellinux_${MP}/work.pcl
        @ echo '../netcdf/intellinux/work.pc' >> intellinux_${MP}/work.pcl
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=intellinux\
        FC=ifc FC90=ifc F90FF='-FR' MPIFC='${INTELLINUX_MPIDIR}/bin/mpif90 -f90=ifc' MPIDIR=${INTELLINUX_MPIDIR} \
        FFLAGS="${INTELLINUX_OPT} ${INTELLINUX_FFLAGS} ${INTELLINUX_DEFINES}" \
        DEFINES="${INTELLINUX_DEFINES}" \
        LIBS= ${LIBS_intellinux} \
        INCLUDES='-I${SRCDIR} -I../netcdf/intellinux -I../netcdf' \
        EXTRABINS='fargc.o'                                                                

# ------------------- Sun Microsystems -----------------------------
# Assuming Sun Fortran version 4.0 (?) with Ultra-SPARC features:
SUN_OPT = -dalign -g -Qoption fpp -P

LIBS_sun_ultra=-xlic_lib=sunperf -xlibmopt -L/gbar/bohr/home1/fys/fyslbh/fftw/lib -lfftw
LIBS_sun_ultra_mpi= -xlic_lib=sunperf -xlibmopt -L/opt/SUNWhpc/lib -lmpi

ULTRA_DEFINES = -DSUN -DULTRA  ${USEFFTW} ${QRAD_SWAP_OPT}
sun_ultra:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=sun_ultra FC=f90 \
        FC90=f90  F90FF=' -free ' MPIFC=/opt/SUNWhpc/bin/mpf90  FFLAGS='${SUN_OPT} ${ULTRA_DEFINES}' \
        DEFINES="${ULTRA_DEFINES}" \
        INCLUDES='-I${SRCDIR} -I../netcdf -M../netcdf/sun_ultra' \
        MPIDIR='/opt/SUNWhpc'
        # EXTRABINS='../binobj/sun_ultra/zfft.o' MPIDIR='/opt/SUNWhpc'

SS10_DEFINES = -DSUN -DSS10
sun_ss10:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=sun_ss10 FC=f90 \
        FC90=f90    F90FF=' -freeform ' MPIFC=f90 \
        FFLAGS="${SUN_OPT}  -depend ${SS10_DEFINES} -xtarget=ss10/hs11 \
                -Qoption fpp -P -xarch=v8" \
        DEFINES="${SS10_DEFINES}" \
        INCLUDES='-I${SRCDIR} -I../netcdf -M../netcdf/sun_ultra'  \
        EXTRABINS=hyperfft.o                                                                                

# ------------------- IBM RS/6000 ----------------------------------
ibm_power3:    
        @ ${MAKE} -f ${MAKEFILE} ibm MP=${MP} ARCH=$@ \
                DEFINES='-DIBM ${QRAD_SWAP_OPT}' QARCH='-qarch=pwr3 -qtune=pwr3'

ibm_power2_wide:    
        @ ${MAKE} -f ${MAKEFILE} ibm MP=${MP} ARCH=$@ \
                DEFINES='-DIBM_POWER2_WIDE  ${QRAD_SWAP_OPT}' QARCH='-qarch=pwr2'

ibm_power2_thin:    
        @ ${MAKE} -f ${MAKEFILE} ibm MP=${MP} ARCH=$@ \
                DEFINES='-DIBM_POWER2_THIN  ${QRAD_SWAP_OPT}' QARCH='-qarch=pwr2 -qtune=pwr2s'

ibm_power:    
        @ ${MAKE} -f ${MAKEFILE} ibm MP=${MP} ARCH=$@ \
                DEFINES='-DIBM_POWER' QARCH='-qarch=pwr'

ibm_ppc:    
        @ ${MAKE} -f ${MAKEFILE} ibm MP=${MP} ARCH=$@ \
                DEFINES='-DIBM_PPC' QARCH='-qarch=ppc'

# IBM XLF version 3.2 compiler flags:

# Checking procedure interface and common blocks
# XLF_CHECK = -qextchk
# Listing and cross-reference
# XLF_CROSSREF = -qxref=full -qattr
# XLF_LISTING = -qsource -qlistopt ${XLF_CROSSREF}
# Compiling for "gprof" performance profiling:
# XLF_PROFILE = -pg
# Compiling for "tprof" performance profiling:
# XLF_PROFILE = -qlist -g -d -bnoquiet
# Debug code:
# XLF_DEBUG = -WF,-DDEBUG -WF,-DALL_RESIDUALS  -WF,-DDEBUG3 -WF,-DDEBUG2
# XLF_DEBUG = -WF,-DALL_RESIDUALS   -WF,-DREALPROJ
XLF_DEBUG = -WF,-DALL_RESIDUALS 
# Floating-point optimizations:

# flags for debugging the code
# XLF_OPT = -g -d -C  -qspillsize=32000 -qfloat=fltint:rsqrt:hssngl -qalias=noaryovrlp:std:nointptr -qdpc=e -qnosave -bmaxdata:0x800
00000 -bmaxstack:0x80000000

# optimizing flags for jensen (large memory)
# XLF_OPT = -d -O3  -qspillsize -qfloat=fltint:rsqrt:hssngl -qalias=noaryovrlp:std:nointptr -qdpc=e -qnosave  -bmaxdata:0x80000000 -
bmaxstack:0x80000000

# optimizing flags for camp 
# On Power3 and PowerPC omit the "hssngl" option
XLF_OPT = -O2 -d -qspillsize -qfloat=fltint:rsqrt:hssngl -qalias=noaryovrlp:std:nointptr -qdpc=e -qnosave

# Special case for F90-modules on the AIX GPFS filesystem
# (see http://hpcf.nersc.gov/computers/SP/f90modules.html)
# XLF_GPFS = -qmoddir=$(TMPDIR) -I$(TMPDIR) 

XLF_FLAGS = ${XLF_OPT} ${XLF_PROFILE} ${XLF_LISTING} ${XLF_DEBUG} ${XLF_GPFS}
# Define ESSL if you have the IBM ESSL library installed:
#ESSL = -DESSL 

# XLF special: prepend "-WF," to the -D flags: 
PREPEND_WF = 's/[       ]*-D/ -WF,-D/g' 
LIBS_ibm_power2_thin=-lesslp2 -lmass
LIBS_ibm_power2_wide=-lesslp2 -lmass
LIBS_ibm_power3= -L/home/walkup/lib -lessln -lmass

#AN CHANGE -- trying to use blas instead of essl

#LIBS_bm_power3= -L/home/anilekar/Installs/Libs_related_to_Dacapo/ -lblas 

ibm:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=${ARCH} \
        FC=xlf FC90=xlf90 MPIFC=mpxlf MPIDIR=/usr/lpp/ppe.poe \
        FFLAGS="${QARCH} ${XLF_FLAGS} \
                `echo ${ESSL} ${DEFINES} | sed ${PREPEND_WF} `" \
        DEFINES='${ESSL} ${DEFINES}' \
        INCLUDES='-I${SRCDIR} -WF,-I${SRCDIR} ${NETCDFINC} -WF,${NETCDFINC}' CPP_PREFIX=-WF, \
        INLINE=-Q+exchns,corlns,corgns,gcorns,EXCH,CORLSD,GCOR,CORGGA,exch1,cor1lsd,gcor1,cor1gga,cor2gga

# ---------------------- CRAY UNICOS ---------------------
# CRAY_LISTING = -rm
# CRAY_OPT = -O3,inline2 -dp
CRAY_OPT = -dp
CRAY_DEFINES = -DCRAY -DBLAS

cray:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=unicos \
        FC=f90 \
        FFLAGS="${CRAY_OPT} ${CRAY_LISTING} -F ${CRAY_DEFINES}" \
        DEFINES="${CRAY_DEFINES}" \
        INCLUDES='-I ${SRCDIR}'
#       INLINE=-Q+exchns,corlns,corgns,gcorns,EXCH,CORLSD,GCOR,CORGGA,exch1,cor1lsd,gcor1,cor1gga,cor2gga

# ------------------- SGI n32 ABI ----------------------------------
SGI_DEFINES = -DSGI -DBLAS
#SGI_OPT = -mips4 -Ofast=ip27 -n32 -r10000 -IPA:aggr_cprop:echo=ON -OPT:IEEE_arithmetic=3
#SGI_OPT = -g -64 -r10000 -static
#SGI_OPT = -g -n32 -r10000
# SGI_OPT = -Ofast=ip27 -64 -r10000 -IPA:aggr_cprop:echo=ON -OPT:IEEE_arithmetic=3
SGI_OPT = -C -g -64 -r10000 -OPT:IEEE_arithmetic=3 -MP:check_reshape=on
LIBS_sgi= -lcomplib.sgimath -lblas -lfastm -lm
LIBS_sgi_mpi= -lmpi
CPP_sgi= /usr/lib/cpp -P ${SGI_DEFINES} 

sgi:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=sgi FC=f90 \
        FC90=f90 F90FF='-freeform' MPIFC=f90           \
        FFLAGS="-cpp -Wp,-P ${SGI_OPT} ${SGI_DEFINES}" \
        DEFINES="${SGI_DEFINES} " \
        INCLUDES='-I${SRCDIR} -I../netcdf -I../netcdf/sgi' \
        INLINE=-INLINE:must=exchns,corlns,corgns,gcorns,EXCH,CORLSD,GCOR,CORGGA,exch1,cor1lsd,gcor1,cor1gga,cor2gga

# ---------------------- Digital Alpha ---------------------
ALPHA_DEFINES = -DDXML -DDECALPHA ${QRAD_SWAP_OPT} ${USEFFTW}
# ALPHA_DEFINES = -DDXML -DDECALPHA ${QRAD_SWAP_OPT}
# Warning: Must compile ms.F with -O0 or -g (bug in DEC Fortran 5.1)
NOOPT= -O0 -g
NOOPT= -g
# ALPHA_OPT = -O4 -arch host -fast -tune host -transform_loops -d
# V5.3-1120 need -switch noexpand_complex to handle optimization
ALPHA_OPT = -O4 -arch host -fast -tune host -transform_loops -d -switch noexpand_complex
# ALPHA_OPT = -O4 -arch host -tune host -d
# ALPHA_OPT = -C -g -d  -K
# ALPHA_OPT = -g -arch host 
ALPHA_FFLAGS = -automatic -convert big_endian -assume byterecl -warn truncated_source
LIBS_alpha=-ldxml -lfftw
# LIBS_alpha=-ldxml

alpha:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=alpha\
        FC=f90 FC90=f90 F90FF='-free' MPIFC=mpif90 MPIDIR=/usr/local/mpi \
        FFLAGS="-cpp -Wp,-P ${ALPHA_OPT} ${ALPHA_FFLAGS} ${ALPHA_DEFINES}" \
        DEFINES="${ALPHA_DEFINES}" \
        LIBS= ${LIBS_alpha} \
        INCLUDES='-I${SRCDIR} -I../netcdf/alpha -I../netcdf' \

# ------------------- Fujitsu VPP-500 vector-parallel --------------
# You may find GNU make in /usr/local/parmacs/bin/fujitsu-vpp500/gnumake
# if the PARMACS library has been installed.
#
# -Oe,-P for optimization without possible evaluation of statements
#        beyond unfulfilled if-statement
# -Ad to promote real*4 to real*8 but not real*8 to real*16
# -Ab to allow for arrays being undefined in intermediate calling levels
VPP_OPT = -Oe,-P -Ab
# The last item (,100) is the max. number of statements
VPP_INLINE = -Ne,exchns,corlns,corgns,gcorns,EXCH,CORLSD,GCOR,CORGGA,exch1,cor1lsd,gcor1,cor1gga,cor2gga,100
# -Wv,-m2 to add to listing the reasons for non-vectorization
# Procedure call debugging:
#  -Da : argument validation check
#         (argument consistency between caller and called subroutine)
#  -Ds : subscript validation(array range) check
#  -Du : undefined variable check
# VPP_DEBUG = -Dasu
# -Pdlt for getting list
VPP_LISTING = -Z$(@:.o=.list) -Pdilt -Wv,-m2 -Eielpu

VPP_FFTLIB = /home/vpp/group12/hammer/lib/libfft.a

# This is for compiling into "new type loadmodule" that works also on 1 GB nodes
VPP_NEW1GB = -Wl,-J,-K
VPP_LDFLAGS = -J -K

VPP_FFLAGS = ${VPP_OPT} ${VPP_NEW1GB} ${VPP_LISTING} ${VPP_DEBUG}

vpp500:
        @ ${MAKE} -f ${MAKEFILE} ${MP} ARCH=vpp500 FC='frtpx' \
        FFLAGS="${VPP_FFLAGS}" \
        DEFINES='-DVPP500' CFLAGS='-DVPP500' \
        INCLUDES=-I${SRCDIR} \
        INLINE="${VPP_INLINE}" \
        EXTRABINS="fft3dj.o machjp.o hostname.o ${VPP_FFTLIB}"

#####################################################################
#
# Rules for serial or message-passing versions
#

# Copy the Makefile to the OBJECTDIR:
${OBJECTDIR}/${MAKEFILE}: ${MAKEFILE}
#       Catch errors: first make sure that ARCH is set.
        @if [ -z "${ARCH}" ]; then \
            echo Error: ARCH not specified; \
            ${MAKE} -f ${MAKEFILE} default; \
            exit 1; \
        fi
        @if [ ! -d ${OBJECTDIR} ]; then \
            ${MKDIR} ${OBJECTDIR}; \
        fi
        cp -p $< ${OBJECTDIR}

serial: ${OBJECTDIR}/${MAKEFILE}
        @echo
        @echo Making $@ version in directory ${OBJECTDIR}
        @echo
        @(cd ${OBJECTDIR}; ${MAKE} -f ${MAKEFILE} dacapo.run \
        FC=${FC} FFLAGS='${FFLAGS}' DEFINES='${DEFINES}' \
        INCLUDES='${INCLUDES}' LIBS='${LIBS}' \
        ALLBINS='${SERIALBINS} ${EXTRABINS}' \
        INLINE='${INLINE}'  )

mpi: ${OBJECTDIR}/${MAKEFILE}
        @echo
        @echo Making $@ version in directory ${OBJECTDIR}
        @echo
                @(cd ${OBJECTDIR}; ${MAKE} -f ${MAKEFILE} dacapo.run \
                LIBS='${LIBS}'  \
                FC='${MPIFC}' \
                FFLAGS='${FFLAGS} ${CPP_PREFIX}-DMPI \
                        -I${MPIDIR}/include ${CPP_PREFIX}-I${MPIDIR}/include' \
                DEFINES='${DEFINES} -DMPI' INCLUDES='${INCLUDES}' \
                ALLBINS='${PARALLELBINS} ${EXTRABINS}' \
                INLINE=${INLINE} )

#####################################################################
#
# Auxiliary libraries
#

# LAPACK

# The library in the destination directory
${DESTDIR}/liblapack.a: ${SRCDIR}/lapack/${ARCH}/liblapack.a
        @if [ ! -d ${DESTDIR} ]; then \
           ${MKDIR} ${DESTDIR}; \
        fi
        cp -p ${SRCDIR}/lapack/${ARCH}/liblapack.a $@

${SRCDIR}/lapack/${ARCH}/liblapack.a: FRC
        @echo
        @echo Making $@ in directory ${SRCDIR}/lapack/${ARCH}
        @echo
        @if [ ! -d ${SRCDIR}/lapack/${ARCH} ]; then \
                ${MKDIR} ${SRCDIR}/lapack/${ARCH}; \
        fi
        @(cd ${SRCDIR}/lapack/${ARCH}; \
                ${MAKE} -f ../Makefile liblapack.a FC='${FC}' FFLAGS='${FFLAGS}' )

# BLAS

# The library in the destination directory
${DESTDIR}/libmyblas.a: ${SRCDIR}/blas/${ARCH}/libmyblas.a
        @if [ ! -d ${DESTDIR} ]; then \
           ${MKDIR} ${DESTDIR}; \
        fi
        cp -p ${SRCDIR}/blas/${ARCH}/libmyblas.a $@

${SRCDIR}/blas/${ARCH}/libmyblas.a: FRC
        @echo
        @echo Making $@ in directory ${SRCDIR}/blas/${ARCH}
        @echo
        @if [ ! -d ${SRCDIR}/blas/${ARCH} ]; then \
           ${MKDIR} ${SRCDIR}/blas/${ARCH}; \
        fi
        @(cd ${SRCDIR}/blas/${ARCH}; \
        ${MAKE} -f ../Makefile libmyblas.a FC='${FC}' FFLAGS='${FFLAGS}' )

# --- netCDF      
${SRCDIR}/netcdf/${ARCH}/netcdfinterface.o : FRC
        @echo
        @echo Making $@ in directory ${SRCDIR}/netcdf/${ARCH}
        @echo
        @if [ ! -d ${SRCDIR}/netcdf/${ARCH} ]; then \
           ${MKDIR} ${SRCDIR}/netcdf/${ARCH}; \
        fi
        @(cd ${SRCDIR}/netcdf/${ARCH} ; \
        ${MAKE} -f ../Makefile netcdfsupport FC90=${FC90} FFLAGS='${FFLAGS}' CPP='${CPP_${ARCH}}' )
 
FRC:
#####################################################################
#
# Fundamental rules for libraries and .o files
#

# Dependency search path (GNU make special):
VPATH = ${SRCDIR}

# General rule for making .o files:
.F.o:
        ${FC90} -c ${FFLAGS} ${F90FF} ${INCLUDES} $<

# Special cases for fortran90 modules, until all
# sources are converted to free format F90

run_context.o :   run_context.F ${SRCDIR}/netcdf/${ARCH}/netcdfinterface.o basicdata.o

generate_kpoints.o :   generate_kpoints.F ${SRCDIR}/netcdf/${ARCH}/netcdfinterface.o basicdata.o
 
read_structure_netcdf.o : read_structure_netcdf.F real_nonlocal.o  ${SRCDIR}/netcdf/${ARCH}/netcdfinterface.o \
                          matrix_utilities.o 

real_nonlocal.o :   real_nonlocal.F  basicdata.o  

setuop.o: setuop.F ${SRCDIR}/netcdf/${ARCH}/netcdfinterface.o basicdata.F basicdata.o

# Special cases for those modules that have inlined subroutines:
pw.o: pw.F definitions.h
        ${FC90} -c ${FFLAGS} ${F90FF} ${INLINE} ${INCLUDES} $<

pwunsp.o: pwunsp.F definitions.h
        ${FC90} -c ${FFLAGS} ${F90FF} ${INLINE} ${INCLUDES} $<

qrutil.o: qrutil.F definitions.h
        ${FC90} -c ${FFLAGS} ${F90FF} ${INLINE} ${INCLUDES} $<

# Special cases for parallel routines, which should be compiled without 
# optimization on alpha 
ms.o: ms.F definitions.h
        ${FC90} -c ${FFLAGS} ${F90FF} ${NOOPT} ${INLINE} ${INCLUDES} $<       
par_getwf.o: par_getwf.F definitions.h
        ${FC90} -c ${FFLAGS} ${F90FF} ${NOOPT} ${INLINE} ${INCLUDES} $<       
par_functions.o: par_functions.F definitions.h
        ${FC90} -c ${FFLAGS} ${F90FF} ${NOOPT} ${INLINE} ${INCLUDES} $< 

definitions.h: timers.h precision.h ibm.h msgpass.h

dacapo.run: ${ALLBINS}  ${SRCDIR}/lapack/${ARCH}/liblapack.a  ${SRCDIR}/blas/${ARCH}/libmyblas.a \
            ${SRCDIR}/netcdf/${ARCH}/netcdfinterface.o ${SRCDIR}/netcdf/${ARCH}/MakeRestrictedSetCopy.o
        ${FC} ${FFLAGS} -o dacapo.run ../netcdf/${ARCH}/netcdfinterface.o ${SRCDIR}/netcdf/${ARCH}/MakeRestrictedSetCopy.o  ${SRCDIR
}/${OBJECTDIR}/basicdata.o ${ALLBINS} ${SRCDIR}/lapack/${ARCH}/liblapack.a  ${SRCDIR}/blas/${ARCH}/libmyblas.a ${LIBS} ../netcdf/${A
RCH}/libnetcdf.a  ${LIBS_${ARCH}}  ${LIBS_${ARCH}_${MP}}

clean.f:
        rm -f *.f

# Dont remove *.i -- we have one as a source include file!
clean:
        rm -f dacapo *.o core out fort.*