CHPC Software: MPI

Message Passing Interface (MPI) is a principal method of performing parallel computations on all CHPC clusters. Its main component is a standardized library that enables communication between processors in distributed processor environments. There are numerous MPI distributions available and thus CHPC supports only some of them, those we believe are best suited for the particular system.

More information: MPI standard page.

Notes on installation on the CHPC machines:

Arches

Since Arches subclusters provide three versions of network interconnects, there are three types of MPI available, MPICH (or equivalent) for MPI communication over TCP/IP (default on tunnelarch and marchingmen), MPICH-MX for communication over Myrinet (default on delicatearch) and MVAPICH (or equivalent) or OpenMPI for InfiniBand (default on sanddunearch.

Two major versions of MPICH are available for TCP/IP. MPICH is an old time favorite release from Argonne NL supporting MPI standard 1.1. MPICH2 is a successor to MPICH supporting MPI 2.0 and offering superior performance over MPICH, especially in collective communication functions.

MVAPICH InfiniBand MPI distribution on Sanddunearch are also based on MPICHes, and as such, there is MVAPICH, which is derived from MPICH, and MVAPICH2, which is derived from MPICH2. Furthermore, Sanddunearch also offers OpenMPI, which performance-wise is slightly worse than the MVAPICHes, but, promises improvements in the future.

Finally, since we have had problems linking GNU compiled MPICH with PGI Fortran compilers, there are two versions of each MPI distribution (= MPICH, MPICH2, MPICH-MX, MVAPICH, MVAPICH2), one compiled with GNU compilers, which is a default, and one compiled with PGI compilers.

MVAPICH

MVAPICH is a port of the open source MPICH distribution targeted for InfinBand interconnect network. It is developed at Network-Based Computing Laboratory at Ohio State University. Note that MVAPICH is installed on sanddunearch only.

Current version of MVAPICH installed on Arches is 0.9.8 It has full MPI-1.1 standard support and certain parts of MPI-2 standard support (MPI-I/O, additional data types and C++ function bindings).

GNU/Pathscale MVAPICH

MVAPICHX distribution compiled with GNU compilers is located in:

/uufs/sanddunearch.arches/sys/pkg/mvapich/std

To use MVAPICH on sanddunearch, source the following shell scripts, which set the correct executable and dynamic library path:

  • source /uufs/sanddunearch.arches/sys/pkg/mvapich/std/etc/mvapich.csh (for csh/tcsh)
  • source /uufs/sanddunearch.arches/sys/pkg/mvapich/std/etc/mvapich.sh (for sh/bash)

One can then uses plain compiler names, such as mpicc,mpicxx, mpif77,mpipathcc,mpipathCC or mpif90 (using pathf90).

Alternatively, full path to MVAPICH scripts with GNU compilers are:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpif77
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpicc
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpicxx

For compilation with Pathscale compilers, use these scripts:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpif90
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpipathcc
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpipathCC

Make sure to source Pathscale shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)

To run, provided you sourced the MVAPICH shell script, just run:

  • mpirun_rsh -rsh -np $PROCS -hostfile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

Alternatively, one can specify the full path, but, then, make sure that the OFED libraries (InfiniBand driver) are in the dynamic library search path: :

  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std/bin/mpirun_rsh -rsh -np $PROCS -hostfile $PBS_NODEFILE ./executable

Important note: Due to system limitation, standard launch of MVAPICH job via mpirun_rsh command will fail if using more than about 230 CPUs. For larger jobs (> 64 CPUs), we recommend using MVAPICH2, which has a different and more efficient launch strategy (see below).

PGI MVAPICH

For users in need to compile MPI code with PGI compilers, we offer MVAPICH distribution compiled exclusively with PGI compilers, located in:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi

To use MVAPICH with PGI, source one of shell scripts below :

  • source /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/etc/mvapich.csh (for csh/tcsh)
  • source /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/etc/mvapich.sh (for sh/bash)

One can then use just the plain compiler names, such as mpicc,mpicxx, mpif77 or mpif90.

Another option that prevents possible mixup with default MPI distribution on Sanddunearch is to use direct paths to the MVAPICH scripts:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/bin/mpif77
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/bin/mpif90
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/bin/mpicc
  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/bin/mpicxx

Make sure to source PGI shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.sh (for sh/bash)

To run the MVAPICH executable, either in batch script or interactive batch:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich/std_pgi/bin/mpirun_rsh -rsh -np $PROCS -hostfile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

MVAPICH2

MVAPICH2 is an open source implementation of MPI 2.0 standard developed at Ohio State University, based on MPICH2 from Argonne NL. Its current release, 0.9.8, is installed on sanddunearch. It performs similarly or slightly better than MVAPICH or OpenMPI. With respect to MVAPICH, it also allows launch of larger jobs using different launch mechanism. Therefore, we recommend MVAPICH2 as the first choice of MPI on Sanddunearch, and, it is set as a default.

GNU/Pathscale MVAPICH2

MVAPICH2 distribution compiled with GNU compilers is located in:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich2/std

MVAPICH2 is usually set as a default, alternatively, user can source the following in his/her shell script. Note that this shell init script needs to be sourced so that MVAPICH2 works correctly.

  • source /uufs/sanddunearch.arches/sys/pkg/mvapich2/std/etc/mvapich2.csh (for csh/tcsh)
  • source /uufs/sanddunearch.arches/sys/pkg/mvapich2/std/etc/mvapich2.sh (for sh/bash)

Then use the plain compiler names, such as mpicc, mpicxx, mpif77, mpipathcc, mpipathCC or mpif90 (for pathf90) to compile.

In case Pathscale compilers are used, make sure to source Pathscale shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)
PGI MVAPICH2

MVAPICH2 distribution compiled PGI C, C++ and Fortran is located in:

  • /uufs/sanddunearch.arches/sys/pkg/mvapich2/std_pgi

Before compiling, one must set up the environment for PGI compilers:

  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.sh (for sh/bash)

In order to use MVAPICH2, source the following shell script:

  • source /uufs/sanddunearch.arches/sys/pkg/mvapich2/std_pgi/etc/mvapich2.csh (for csh/tcsh)
  • source /uufs/sanddunearch.arches/sys/pkg/mvapich2/std_pgi/etc/mvapich2.sh (for sh/bash)

One then can use just the plain compiler names, such as mpicc, mpicxx, mpif77 or mpif90 to compile.

Running MVAPICH2 programs

There are several steps to be taken in order to run MVAPICH2 executable. The reason for this is that MVAPICH2, like MPICH2, uses daemon-based process startup. So, in order to run MVAPICH2 program, the daemons first must be started on the selected nodes, and, only then the program can execute.

First time users of either MPICH2 or MVAPICH2 must create file .mpd.conf in their home root directory with this line in it: secretword=my_code. Do not use my_code, change it to password of your liking. Also, make sure that .mpd.conf can be only read by you by chmod 600 .mpd.conf

Then it's a good idea to make sure that all works:

  1. qsub -I into the nodes, e.g. qsub -I -l nodes=2,walltime=0:10:00
  2. Start MVAPICH2 MPD, e.g. mpdboot -n <num-of-nodes> -r rsh -f $PBS_NODEFILE
  3. Test the MPD ring with mpdtrace, the output should be node names one per line.
  4. Test something with mpdrun, e.g. mpdrun -np 2 hostname
  5. Run quick MPI test program, e.g. mpiexec -n 2 ./test.

If all test are successful, you can set up your PBS script to run MVAPICH2 as:

.....
cd $WORK_DIR
mpdboot -n $NODES -r /usr/bin/rsh -f $PBS_NODEFILE
mpiexec -n $PROCS ./executable

Note that $NODES = $PROCS/2 on dual processor nodes, or $NODES = $PROCS/4 on dual-core dual processor nodes (Sanddunearch, part of Landscapearch). Make sure that you set it up like this in your script otherwise the mpdboot command will fail.

OpenMPI

OpenMPI is an effort combining several previously available MPI distributions to produce the best available MPI library. It is MPI-2 compliant, and, has many appealing features. However, our tests show that for the InfiniBand, its performance is slightly below that of MVAPICH and MVAPICH2. Also, our tests suggest that the Myrinet port of OpenMPI is slower than MPICH-MX. As such, OpenMPI is currently installed only on sanddunearch for GNU/Pathscale compilers only. However, we keep an eye on the new developments with OpenMPI, and, will most likely expand its support in the future.

Current version of OpenMPI installed on Arches is 1.2. For details of its features, see OpenMPI webpage.

GNU/Pathscale OpenMPI

OpenMPI distribution compiled with GNU compilers is located in:

/uufs/sanddunearch.arches/sys/pkg/openmpi/std

To use OpenMPI, source the following shell script. This script sets executable and dynamic library paths, without which OpenMPI will not work correctly.

  • source /uufs/sanddunearch.arches/sys/pkg/openmpi/std/etc/openmpi.csh (for csh/tcsh)
  • source /uufs/sanddunearch.arches/sys/pkg/openmpi/std/etc/openmpi.sh (for sh/bash)

Then use the plain compiler names, such as mpicc, mpicxx, mpif77, or mpif90 to compile.

Make sure to source Pathscale shell scripts before compiling and running executables compiled with mpif90 :

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)

To run the OpenMPI executable, either in batch script or interactive batch:

  • mpirun -np $PROCS -hostfile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

Note: Please, contact CHPC if you would like to have OpenMPI built with other compilers and on other systems.

MPICH-MX

MPICH-MX is a port by Myricom of the open source MPICH distribution targeted for Myrinet interconnect network. Note that MPICH-MX is installed on delicatearch and landscapearch. It is the default only on delicatearch (symlinked to /uufs/delicatearch.arches/sys/bin). On landscapearch, one has to use the full path to the MPICH-MX distribution, as detailed below. The reason for this is that not all landscapearch nodes have Myrinet. If you use landscapearch and are not sure if your nodes have Myrinet, contact CHPC.

Current version of MPICH-MX installed on Arches is 1.2.7..3 It has full MPI-1.1 standard support and certain parts of MPI-2 standard support (MPI-I/O, additional data types and C++ function bindings).

GNU/Pathscale MPICH-MX

MPICH-MX distribution compiled with GNU compilers is located in:

/uufs/$UUFSCELL/sys/pkg/mpich-mx/std

where $UUFSCELL is either delicatearch.arches or landscapearch.arches.

The best way to use MPICH-MX on delicatearch or landscapearch is to add the bin directory of the distribution to the default path, i.e.:

  • setenv PATH "/uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin:$PATH" (for csh/tcsh)
  • export PATH="/uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin:$PATH" (for sh/bash).

One then uses plain compiler names, such as mpicc,mpicxx, mpif77,mpipathcc,mpipathCC or mpipathf90.

Alternatively, full path to MPICH-MX scripts with GNU compilers are:

  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpif77
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpicc
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpicxx

For compilation with Pathscale compilers, use these scripts:

  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpif90
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpipathcc
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpipathCC

Make sure to source Pathscale shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)

To run the MPICH-MX executable, either in batch script or interactive batch:

  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpirun.ch_mx -np $PROCS -machinefile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

If you sourced the MPICH-MX in your path, just run:

  • mpirun.ch_mx -np $PROCS -machinefile $PBS_NODEFILE ./executable

Important note: Due to system limitation, standard launch of MPICH-MX job via mpirun.ch_mx command will fail if using more than about 230 CPUs. There is an alternative launch strategy that is more efficient for larger jobs and should be used when running on more than 64 CPUs. This strategy is turned on by using mpirun.ch_mx option . Therefore, in order to run an MPICH-MX job larger than 64 CPUs, please, use command like this:

/uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpirun.ch_mx --mx-tree-spawn -np $PROCS -machinefile $PBS_NODEFILE ./executable

PGI MPICH-MX

For users in need to compile MPI code with PGI compilers, we offer MPICH-MX distribution compiled exclusively with PGI compilers, located in:

  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi

Since this is not the default MPICH-MX distribution, we don't recommend to add it to default path, but, if the users insists, he can add the following to $PATH and bear consequences of possible mix-up:

  • setenv "/uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi/bin:$PATH" (for csh/tcsh)
  • export "/uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi/bin:$PATH" (for sh/bash)

One then use just the plain compiler names, such as mpicc,mpicxx, mpif77 or mpif90.

A better way is to use direct paths to the MPICH-MX scripts:

  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi/bin/mpif77
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi/bin/mpif90
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi/bin/mpicc
  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std_pgi/bin/mpicxx

Make sure to source PGI shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.sh (for sh/bash)

To run the MPICH-MX executable, either in batch script or interactive batch:

  • /uufs/$UUFSCELL/sys/pkg/mpich-mx/std/bin/mpirun.ch_mx -np $PROCS -machinefile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

MPICH

MPICH is an open source public domain portable MPI distribution developed at Argonne NL. It is being superceeded by MPICH2, but, is still actively maintained. Arches run the ch_p4 device version of MPICH. This is the default on marchingmen and tunnelarch, and can also be used on delicatearch, although MPICH-MX will most likely provide superior performance there.

Current version of MPICH installed on Arches is 1.2.7. It has full MPI-1.1 standard support and certain parts of MPI-2 standard support (MPI-I/O, additional data types and C++ function bindings).

GNU/Pathscale MPICH

MPICH distribution compiled with GNU compilers is located in:

  • /uufs/arches/sys/pkg/mpich/std

This is the default MPI distribution on marchingmen and tunnelarch (symlinked to /uufs/arches/sys/bin or /uufs/arches/sys/bin), which don't have Myrinet network.

The best way to use MPICH on these two clusters is to add the bin directory of the distribution to the default path via:

  • source /uufs/arches/sys/pkg/mpich/std/etc/mpich.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/mpich/std/etc/mpich.sh (for sh/bash)

One then use just the plain compiler names, such as mpicc,mpicxx, mpif77,mpipathcc,mpipathCC or mpipathf90.

Alternatively, full path to MPICH scripts with GNU compilers are:

  • /uufs/arches/sys/pkg/mpich/std/bin/mpif77
  • /uufs/arches/sys/pkg/mpich/std/bin/mpicc
  • /uufs/arches/sys/pkg/mpich/std/bin/mpicxx

For compilation with Pathscale compilers, use these scripts:

  • /uufs/arches/sys/pkg/mpich/std/bin/mpipathf90
  • /uufs/arches/sys/pkg/mpich/std/bin/mpipathcc
  • /uufs/arches/sys/pkg/mpich/std/bin/mpipathCC

Make sure to source Pathscale shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)

To run the MPICH executable, either in batch script or interactive batch:

  • /uufs/arches/sys/pkg/mpich/std/bin/mpirun -np $PROCS -machinefile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

If you sourced the MPICH in your path, just run:

  • mpirun -np $PROCS -machinefile $PBS_NODEFILE ./executable
PGI MPICH

MPICH distribution compiled exclusively with PGI compilers is located in:

  • /uufs/arches/sys/pkg/mpich/std_pgi

We recommend this distribution if the user needs to compile the code with PGI and run either on marchingmen or on tunnelarch.

We don't recommend sourcing this distro source files, but, if user insists, it can be done via:

  • source /uufs/arches/sys/pkg/mpich/std_pgi/etc/mpich.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/mpich/std_pgi/etc/mpich.sh (for sh/bash)

This means that one can just use MPICH compileri script names, such as mpicc,mpicxx, mpif77 or mpif90. We don't recommend this since this can confuse the user with the default MPICH distribution that is compiled with GNU compilers.

We recommend to use full path to MPICH scripts with PGI compilers:

  • /uufs/arches/sys/pkg/mpich/std_pgi/bin/mpif77
  • /uufs/arches/sys/pkg/mpich/std_pgi/bin/mpif90
  • /uufs/arches/sys/pkg/mpich/std_pgi/bin/mpicc
  • /uufs/arches/sys/pkg/mpich/std_pgi/bin/mpicxx

Make sure to source PGI shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.sh (for sh/bash)

To run the MPICH executable, either in batch script or interactive batch:

  • /uufs/arches/sys/pkg/mpich/std/bin/mpirun -np $PROCS -machinefile $PBS_NODEFILE ./executable

$PROCS stands for number of processors used in the run.

MPICH2

MPICH2 is an open source implementation of MPI 2.0 standard developed at Argonne NL. It's current release, 1.0.3, has been thoroughly tested, it is quite stable and better performing than MPICH, which is why we recommend it. It is installed on marchingmen and tunnelarch, and can also be used on delicatearch, although MPICH-MX will most likely provide superior performance there.

Current version of MPICH2 installed on Arches is 1.0.3. It has full MPI-2.0 standard.

Note that MPICH2 uses a different startup mechanism than MPICH, which is detailed below.

GNU/Pathscale MPICH2

MPICH2 distribution compiled with GNU compilers is located in:

  • /uufs/arches/sys/pkg/mpich2/std

In order to use MPICH2 on any of the Arches clusters, one must source the following in his/her shell script:

  • source /uufs/arches/sys/pkg/mpich2/std/etc/mpich2.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/mpich2/std/etc/mpich2.sh (for sh/bash)

One then can use just the plain compiler names, such as mpicc,mpicxx, mpif77,mpipathcc,mpipathCC or mpipathf90 to compile.

In case Pathscale compilers are used, make sure to source Pathscale shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)
PGI MPICH2

MPICH2 distribution compiled with GNU C/C++ and PGI Fortran is located in:

  • /uufs/arches/sys/pkg/mpich2/std_pgi

We were not able to compile MPICH2 with PGI C compilers

.

Before compiling, one must set up the environment for PGI compilers:

  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pgi/std/etc/pgi.sh (for sh/bash)

In order to use MPICH2 on any of the Arches clusters, one must source the following in his/her shell script:

  • source /uufs/arches/sys/pkg/mpich2/std_pgi/etc/mpich2.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/mpich2/std_pgi/etc/mpich2.sh (for sh/bash)

One then can use just the plain compiler names, such as mpicc,mpicxx, mpif77 or mpif90 to compile.

In case Pathscale compilers are used, make sure to source Pathscale shell scripts before compiling and running executables compiled this way:

  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/pscale/std/etc/pscale.sh (for sh/bash)
Running MPICH2 programs

There are several steps to be taken in order to run MPICH2 executable. The reason for this is that MPICH2 uses daemon-based process startup. So, in order to run MPICH2 program, the daemons first must be started on the selected nodes, and, only then the program can execute.

First time users must create file .mpd.conf in your home root directory with this line in it: secretword=my_code. Do not use my_code, change it to password of your liking. Also, make sure that .mpd.conf can be only read by you by chmod 600 .mpd.conf

Then it's a good idea to make sure that all works:

  1. qsub -I into the nodes, e.g. qsub -I -l nodes=2,walltime=0:10:00
  2. Start MPICH2 MPD, e.g. mpdboot -n <num-of-nodes> -r /usr/bin/rsh -f $PBS_NODEFILE
  3. Test the MPD ring with mpdtrace, the output should be node names one per line.
  4. Test something with mpdrun, e.g. mpdrun -np 2 hostname
  5. Run quick MPI test program.

If all test are successful, you can set up your PBS script to run MPICH2 as:

.....
cd $WORK_DIR
mpdboot -n $NODES -r /usr/bin/rsh -f $PBS_NODEFILE
mpiexec -n $PROCS ./executable

Note that $NODES = $PROCS/2 on dual processor nodes, make sure that you set it up like this in your script otherwise the mpdboot command will fail.

Last Modified: October 06, 2008 @ 21:07:48