Basic linear algebra libraries

• BLAS libraries on Arches

Advanced mathematics libraries

• LAPACK library
• ScaLAPACK library
• PETSc library
• NAG library

There are several different BLAS library versions on Arches, which are summarized below. Note that Arches clusters consist of several different x86-64 processor families and as such, different BLAS versions are more appropriate to different clusters.

AMD Core Math Library (ACML)

ACML is best suited for AMD based processors. These processors are on delicatearch, marchingmen, tunnelarch and most of landscapearch and sanddunearch. If you run on these clusters, we recommend using ACML for BLAS and LAPACK.

Since 2003, AMD is actively developing mathematical library targeted for its AthlonXP and Opteron processors. As of this writing, it includes full optimized BLAS, LAPACK and fast Fourier transform routines. For more information, consult AMD ACML webpage. From our tests and AMD's presentation, BLAS speed is almost as good as that of ATLAS, the FFT routines are several procent slower than FFTW, but keep improving with new releases. Latest version 4.1.0 available is located at /uufs/arches/sys/pkg/acml/std. Note that from version 4.0 up, support for g77 compiler has been dropped. Fortran users who use g77 will need to use version 3.5.0, which is located at /uufs/arches/sys/pkg/acml/3.5.0/gnu64

Compilation instructions:

Fortran

GNU Fortran:

g77 source_name.f -o executable_name -L/uufs/arches/sys/pkg/acml/3.5.0/gnu64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/3.5.0/gnu64/lib

PGI Fortran:

pgf90 (or pgf77) source_name.f -o executable_name -L/uufs/arches/sys/pkg/acml/std/pgi64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/pgi64/lib

Pathscale Fortran:

pathf90 source_name.f -o executable_name -L/uufs/arches/sys/pkg/acml/std/pathscale64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/pathscale64/lib

Intel Fortran:

ifort source_name.f -o executable_name -L/uufs/arches/sys/pkg/acml/std/ifort64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/ifort64/lib

C/C++

gnu C/C++:

gcc (or g++) source_name.f -o executable_name -L/uufs/arches/sys/pkg/acml/std/gfortran64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/gfortran64/lib

PGI C/C++:

pgcc (or pgCC) source_name.f -o executable_name -L/uufs/arches/sys/pkg/acml/std/pgi64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/pgi64/lib

Pathscale C/C++:

pathcc (or pathCC) source_name.f -o executable_name -L/uufs/acml/sys/pkg/atlas/std/pathscale64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/pathscale64/lib

Intel C/C++:

icc (or icpc) source_name.f -o executable_name -L/uufs/acml/sys/pkg/atlas/std/ifort64/lib -lacml -Wl,-rpath=/uufs/arches/sys/pkg/acml/std/ifort64/lib

Intel Math Kernel Library (MKL)

MKL is best suited for Intel based processors. These processors are on updraft, hiddenarch (meteoxx nodes) and on some newer landscapearch and sanddunearch nodes (Simons, Molinero, Schuster).. If you run on these clusters, we recommend using MKL for BLAS and LAPACK. We also recommend using Intel Fortran and C/C++ for best performance.

MKL contains highly optimized math routines. It includes full optimized BLAS, LAPACK, sparse solvers, vector math library, random number generators and and fast Fourier transform routines (including FFTW wrappers). For more information, consult Intel MKL webpage. Latest version 10.0 available is located at /uufs/arches/sys/pkg/mkl/std.

Compilation instructions:

Intel Fortran

ifort source_name.f -o executable_name -L/uufs/arches/sys/pkg/mkl/std/em64t/lib -lmkl -lguide -Wl,-rpath=/uufs/arches/sys/pkg/mkl/std/em64t/lib

Intel C/C++

icc (or icpc) source_name.c -o executable_name -L/uufs/arches/sys/pkg/mkl/std/em64t/lib -lmkl -lguide -Wl,-rpath=/uufs/arches/sys/pkg/mkl/std/em64t/lib

ATLAS

Automatically Tuned Linear Algebra Software (ATLAS) is an open source library aimed at providing portable performance solution. It provides full BLAS and certain LAPACK routines, which are being tuned to the computer platform at the compilation time. This is the BLAS-compatible library that we recommend using. ATLAS has been optimized for the Opteron platform and from our tests achieves the best performance in a set of BLAS operations. The library is located at /uufs/arches/sys/pkg/atlas/std/lib.

The current version is 3.7.2.

Compilation instructions:

Fortran

GNU Fortran:

g77 source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -lf77blas -latlas

To include also the LAPACK subset in ATLAS:

g77 source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -llapack -lcblas -lf77blas -latlas

PGI Fortran:

pgf90 (or pgf77) source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -lpgf90blas (or -lpgf77blas) -latlas

To include also the LAPACK subset in ATLAS:

pgf90 (or pgf77) source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -lpgf90lapack (or -lpgf77lapack) -lcblas -lpgf90blas (or -lpgf77blas) -latlas

Pathscale Fortran:

pathf90 source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -lpathf90blas -latlas

To include also the LAPACK subset in ATLAS:

pathf90 source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -lpathf90lapack -lcblas -lpathf90blas -latlas

C/C++

gnu C/C++:

gcc (or g++) source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -latlas -lcblas

PGI C/C++:

pgcc (or pgCC) source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -latlas -lcblas

Pathscale C/C++:

pathcc (or pathCC) source_name.f -o executable_name -L/uufs/arches/sys/pkg/atlas/std/lib -latlas -lcblas

GOTO library

GOTO library by Kazushige Goto is another high performance implementation of BLAS and partial LAPACK, which performance is comparable with ATLAS. AMD experts claim that GOTO's BLAS3 performance (DGEMM in particular) is slightly better than Atlas. For more information, consult GOTO webpage. Latest GOTO library is located in /uufs/arches/sys/pkg/goto/std/lib.

Compilation instructions:

Fortran

GNU Fortran:

g77 source_name.f -o executable_name -Wl,-rpath=/uufs/arches/sys/pkg/goto/std/lib /uufs/arches/sys/pkg/goto/std/lib/libgoto.so

PGI Fortran:

pgf90 (or pgf77) source_name.f -o executable_name -Wl,-rpath=/uufs/arches/sys/pkg/goto/std/lib /uufs/arches/sys/pkg/goto/std/lib/libgoto.so

Pathscale Fortran:

pathf90 source_name.f -o executable_name -Wl,-rpath=/uufs/arches/sys/pkg/goto/std/lib /uufs/arches/sys/pkg/goto/std/lib/libgoto.so

C/C++

GNU C/C++:

gcc source_name.f -o executable_name -Wl,-rpath=/uufs/arches/sys/pkg/goto/std/lib /uufs/arches/sys/pkg/goto/std/lib/libgoto.so

PGI C/C++:

pgcc (or pgCC) source_name.f -o executable_name -Wl,-rpath=/uufs/arches/sys/pkg/goto/std/lib /uufs/arches/sys/pkg/goto/std/lib/libgoto.so

Pathscale C/C++:

pathcc source_name.f -o executable_name -Wl,-rpath=/uufs/arches/sys/pkg/goto/std/lib /uufs/arches/sys/pkg/goto/std/lib/libgoto.so

Portland Group BLAS

Portland Group ships its own version of BLAS library with its compilers. This is the BLAS that will get linked to your source when you use PG compilers with -lblas option. We discourage using it since it is not highly optimized. The libraries are located at $PGI/linux86/lib.

Compilation instructions (for reference only):

PGI Fortran

pgf90 (or pgf77) source_name.f -o executable_name -lblas

PGI C/C++

pgcc (pgCC) source_name.c -o executable_name -lblas

Updated: July 20, 2004

PETSc (Portable, Extensible Toolkit for Scientific Computation) is a package incorporating parallel data structures and routines to numerically solve partial differential equations and related linear and non-linear problems .

To Use PETSc:

PETSc is a sophisticated package; and fully understanding its usage requires time. However, application programmers can ea sily begin to use PETSc from a high level, and then gradually learn more details according to their needs. Users should read the PETSc Users Manual before attempting to do any serious coding with PETSc!

Getting information on PETSc:

• On-line:
  • Manual pages on all routines including examples:
    • MPI and MPE: http://www-unix.mcs.anl.gov/mpi/www
    • TAO: http://www-fp.mcs.anl.gov/tao/docs
• Troubleshooting:
  • http://www-unix.mcs.anl.gov/petsc/petsc-2/documentation/troubleshooting.html
• PDF:
  • manual.pdf

To Run PETSc Routines

Before using PETSc, the user must first set the environmental variables PETSC_DIR, indicating the full path of the PETSc home directory and PETSC_ARCH, specifying the architecture (e.g., rs6000, solaris, IRIX, etc.) on which PETSc is being used. For SP and Raptor usage of these environmental variables is specified below. These environmental variables can be placed in a .cshrc file.

All PETSc programs use the MPI standard for message-passing communication. Thus, to execute PETSc programs, users must know the procedure for beginning MPI jobs on their selected computer system(s).

Notes on installation on the CHPC machines

ICE Box:

Current version is 2.1.6

Location: /uufs/sp/sys/pkg/petsc/std

Follow these steps when creating a program using PETSc:

1. Set the following two enviromental variables:
   • setenv PETSC_DIR /uufs/icebox/sys/pkg/petsc/std
   • setenv PETSC_ARCH `$PETSC_DIR/bin/petscarch`
2. Pick an example/tutorial program embedded in the $PETSC_DIR/src tree structure, that is closest to the your problem. Consult the PETSc webpage or browse the $PETSC_DIR/src directory to find a good one. Then compile it using the makefiles that come with the examples make sure they compile and run fine, e.g.:
   • debugging version: BOPT=g exnn
   • optimized version: make BOPT=O exnn
3. Run using the MPICH's mpirun via PBS script, e.g.:
   • /uufs/icebox/sys/mpich/bin/mpirun -np 2 -machinefile=$PBS_NODEFILE ./exnn
4. After the example compiles and runs correctly, modify it for your application.

Due to limited demand and high licensing costs, we have discontinued NAG licensing. If you have a need for NAG, please, contact CHPC at issues@chpc.utah.edu. Below is the instructions that we used when our license was active.

Numerical Algorithms Group, mathematical Fortran and C subroutines library.

Below are links for documentation for the appropriate library. We recommend to follow the intructions for the platforms specified below first with one of the numerous examples that are located in the directory tree of each library. Refer to the links below for detailed documentation on the library routines and their use.

• NAG F77 library documentation
• NAG F90 library documentation
• NAG Fortran SMP library documentation
• NAG Fortran Parallel library documentation
• NAG C library documentation

Notes on installation on the CHPC machines

NAG library is located in /uufs/arches/sys/pkg/nag/. The latest Fortran library is in /uufs/arches/sys/pkg/nag/FL, it is Mark 21 for use with GNU, PGI or Pathscale compilers. The latest C library is in /uufs/arches/sys/pkg/nag/CL/std and is designed for use with GNU compilers and compatibles (i.e. most other compilers such as PGI or Pathscale). All x86-64 NAG libraries use AMD's ACML library for high performance BLAS and LAPACK.

The steps to link the NAG libraries to your code are as follows:

1. Source file that defines environment for NAG:
   • source /uufs/arches/sys/pkg/nag/FL/std_comp/etc/nagf.csh (for Fortran library and csh/tcsh)
   • source /uufs/arches/sys/pkg/nag/CL/std/etc/nagc.csh (for C library and csh/tcsh)
   • source /uufs/arches/sys/pkg/nag/FL/std_comp/etc/nagf.sh (for Fortran library and ksh/bash)
   • source /uufs/arches/sys/pkg/nag/CL/std/etc/nagc.sh (for C library and ksh/bash)
where std_comp is either std_gnu for GNU Fortran, std_pgi for PGI Fortran or std_psc for Pathscale Fortran.
2. Compile with selected Fortran compiler, e.g.:
   • g77 -O3 program.f -L/uufs/arches/sys/pkg/nag/FL/std_gnu/lib -lnag -L/uufs/arches/sys/pkg/acml/std/gnu64/lib -lacml -o executable.exe
   • pgf77 -fastsse program.f -L/uufs/arches/sys/pkg/nag/FL/std_pgi/lib -lnag -L/uufs/arches/sys/pkg/acml/std/pgi64/lib -lacml -o executable.exe
   • pgf90 -fastsse program.f -L/uufs/arches/sys/pkg/nag/FL/std_pgi/lib -lnag -L/uufs/arches/sys/pkg/acml/std/pgi64/lib -lacml -o executable.exe
   • pathf90 -Ofast program.f -L/uufs/arches/sys/pkg/nag/FL/std_psc/lib -lnag -L/uufs/arches/sys/pkg/acml/std/pathscale64/lib -lacml -o executable.exe
3. or with choice of C compiler, e.g.:
   • gcc -O3 program.c -I/uufs/arches/sys/pkg/nag/CL/std/include -L/uufs/arches/sys/pkg/nag/CL/std/lib -lnagc -L/uufs/arches/sys/pkg/acml/std/gnu64/lib -lacml -lg2c -lpthread -lm -o executable.exe

Note that for the C library we include path to C header files and also link GNUs Fortran to C library (g2c) and pthread and math (m) libraries.

Also note that the license is also required during the runtime. Additionally, both libnag and libacml are provided in both static and dynamic form, dynamic being the default. In order to get the right license and the right search path to the libraries, one must source the NAG source files in login scripts ( .tcshrc, .bashrc) before running program compiled with NAG.

LAPACK (Linear Algebra PACKage) provides routines for solving systems of simultaneous linear equations, least-squares solutions of linear systems of equations, eigenvalue problems, and singular value problems. It runs on single processor only, for parallel processing use ScaLAPACK.

LAPACK home page

LAPACK is installed on Icebox and Arches in the /uufs/system_name/sys/pkg/lapack/std/lib tree.

The ScaLAPACK (or Scalable LAPACK) library includes a subset of LAPACK routines redesigned for distributed memory MIMD parallel computers. It is currently written in a Single-Program- Multiple-Data style using explicit message passing for interprocessor communication. It assumes matrices are laid out in a two-dimensional block cyclic decomposition.

ScaLAPACK is designed for heterogeneous computing and is portable on any computer that supports MPI.

Like LAPACK, the ScaLAPACK routines are based on block-partitioned algorithms in order to minimize the frequency of data movement between different levels of the memory hierarchy. (For such machines, the memory hierarchy includes the off-processor memory of other processors, in addition to the hierarchy of registers, cache, and local memory on each processor.)

The fundamental building blocks of the ScaLAPACK library are distributed memory versions (PBLAS) of the Level 1, 2 and 3 BLAS, and a set of Basic Linear Algebra Communication Subprograms (BLACS) for communication tasks that arise frequently in parallel linear algebra computations. In the ScaLAPACK routines, all interprocessor communication occurs within the PBLAS and the BLACS. One of the design goals of ScaLAPACK was to have the ScaLAPACK routines resemble their LAPACK equivalents as much as possible.

ScaLAPACK Support:

On-line:

• Documentation for ScaLAPACK including examples: ScaLAPACK Home Page
• Documentation for BLACS Version 1.1, Patch03: BLACS Web Page

Troubleshooting:

If you have a question about ScaLAPACK, we recommend the FAQ: Frequently Asked Questions list. This FAQ contains answers to popular questions that we have been asked by users.

To Run an Example Program Using ScaLAPACK

Several FORTRAN programs calling PBLAS and ScaLAPACK routines are available for sample runs. See the README page for instructions for running an example program.

Here's how to test a ScaLAPACK example:

1. Select and download into a work directory, the FORTRAN example scaex program to test run:
   • http://www.netlib.org/scalapack/examples/scaex.f
2. In the user's work directory, unzip and tar the file:
   • gunzip -c scaex.tgz | tar xvf -
3. Update the Makefile to reference the libraries shown above for compiling and linking the scaex example.
4. Run the Makefile to produce an executable file called xdscaex:
   • make
5. Submit the scaex example on ICE Box:
   • qsub scalapk_ice.script

Once the above example program has run successfully and the output viewed, redo the above steps with the user's program.

Notes on installation on the CHPC machines

The Arches cluster has several installations of ScaLAPACK based on compiler and MPI distribution used.

1. Build of ScaLAPACK 1.7.5 with GNU compilers and MPICH is in /uufs/arches/sys/pkg/scalapack/std
2. Build of ScaLAPACK 1.7.5 with PGI compilers and MPICH is in /uufs/arches/sys/pkg/scalapack/std_pgi
3. Build of ScaLAPACK 1.8.0 with GNU compilers and MPICH2 is in /uufs/arches/sys/pkg/scalapack/std_mpich2

Since linking ScaLAPACK libraries is rather complex, we recommend taking SLMake.inc from appropriate ScaLAPACK build, e.g. /uufs/arches/sys/pkg/scalapack/std and modifying it for appropriate user source files.

For information on CHPC's MPI distributions and how to compile and run MPI programs, see CHPC's MPI Documentation.