CHPC Software: Vampir

TAU / Intel Trace Analyzer Overview

Intel Trace Analyzer (ITA, fka Vampir) is a graphical profiling tool that enables user to analyze time the program spends in calculation, IO, communication,... It enables the user to quickly focus at the appropriate level of detail by zooming into an arbitrary part of the trace and by selecting interesting processes, events, and communications operations.

The analyzed program must generate a trace file, .vtf, This can be done by an open source package TAU (Tuning and Analysis Utilities). User has to link the code with TAU, run it to produce TAU trace files and then convert them to Vampir format. These files are then read by ITA for performance analysis.

Please note that the trace files can become quite large (gigabytes). Therefore, we recommend to run only a small section of the code with tracing enabled, e.g. single iteration, time step, etc. If that is not possible in your code, please, contact CHPC for options how to enable/disable tracing via code instrumentation.

Location of TAU and the Intel Trace Analyzer on CHPC machines and instructions how to use them are below.

Further references:

TAU / Intel Trace Analyzer on Arches

MPI profiling involves two steps. First one must produce instrumented binary that allows timing information collection. On our systems, we use the TAU package for this purpose. Then one runs the instrumented executable and produces trace files that contain the timing information. Finally, these trace files are viewed in a program that lets user analyze the timing information. At CHPC, we use Intel Trace Analyzer for this purpose.

Binary is instrumented by using special TAU compiler wrappers instead of standard MPI compilers. In order to use these wrappers, one has to first source in the TAU environment.

  • source /uufs/arches/sys/pkg/tau/std/etc/tau.csh for csh/tcsh
  • source /uufs/arches/sys/pkg/tau/std/etc/tau.sh for sh/ksh/bash

Then one can either modify the program's Makefile and replace the default compilers with TAU compiler wrappers. Note that we are including TAU's Makefile to define all the TAU make variables.

  • TAUROOTDIR = uufs/arches/sys/pkg/tau/2.15
  • include $(TAUROOTDIR)/include/Makefile
  • F90 = $(TAU_COMPILER) pathf90
  • CC = $(TAU_COMPILER) gcc

Alternatively, one can compile directly using TAU compiler wrapper scripts, tau_f90.sh,tau_cc.sh, tau_cxx.sh.

Once the executable is compiled, run it to produce the trace files. Note that since this is an MPI program, it must be run with the mpirun command, e.g.:

/uufs/arches/sys/pkg/mpich/std/bin/mpirun -np 4 -machinefile $PBS_NODEFILE ./executable

Upon finishing, there should be numerous files named tautrace.* and events.* in the run directory. These are the trace files in TAU format.

We have to convert these files to the Vampir trace file (vtf) format. This is done in two steps.

  1. tau_merge tautrace.*.trc myprogram.trc - optionally add -n to break a stuck session
  2. tau2vtf myprogram.trc tau.edf myprogram.vtf

Trace files are viewed with the Trace Analyzer. In order to use ITA, source a script in the .cshrc, .tcshrc or .bashrc that sets paths and license information:

  • source /uufs/arches/sys/pkg/ita/std/etc/ita.csh (for csh/tcsh)
  • source /uufs/arches/sys/pkg/ita/std/etc/ita.sh (for ksh/bash)

Then open the Trace Analyzer with the trace file:

traceanalyzer executable.vtf

Finally, here is an example of a Makefile entry for compiling MPEVB extension to DLPOLY molecular dynamics package with TAU.

Initial definitions remain unchanged
STRESS=STRESS
TYPE=3pt

TAUROOTDIR = /uufs/arches/sys/pkg/tau/std
include $(TAUROOTDIR)/include/Makefile
F90 = $(TAU_COMPILER) pathf90

.........
lots of other stuff
.........
arches-pa-tau : dpp cp $(MPI_DIR)/include/mpif.h mpif.h $(MAKE) \ LDFLAGS="-O3 -OPT:Ofast -OPT:Olimit=0 -L$(FFTW_LIBRARY)/lib -lfftw3 " \ FFLAGS="-c -O3 -OPT:Ofast -OPT:Olimit=0 " \ CPFLAGS="-D$(STRESS) -DMPI -DFFTW -P -D'pointer=integer*8' " \ TIMER="" EX=$(EX) BINROOT=$(BINROOT) $(TYPE)
Last Modified: October 06, 2008 @ 21:07:46