As of January 2017, CHPC has obtained a license of the Distributed Debugging Tool (DDT) from Allinea Software and will be maintaining that as compared to the Totalview debugger, which maintenance we have dropped. You can keep using the Totalview but it will be limited to the latest 2016 update. This documentation page lists both DDT and Totalview information, although Totalview documentation will not be updated.
Allinea Distributed Debugging Tool (DDT) is a multi-process, multi-thread graphical debugger for C, C++, Fortran, assembler and mixed source codes with GPU and accelerator support. Its feature set is similar to Totalview, although the user interface is different and more modern.
More information
Allinea was purchased by ARM in late 2016, which has commited to continuous development and support of Allinea tools. For more information on DDT, see the Allinea DDT information.
For details on every aspect of DDT use see the DDT User Guide.
Allinea also has a few introductory videos.
Differences from Totalview
Among the advantages to Totalview are more modern GUI with syntax highlighting with program checking, built in editor and version control.
The basic use of DDT is fairly similar to Totalview, though variable examination is different, especially with arrays. See Viewing Variables and Data from DDT user's guide for details.
DDT is installed in /uufs/chpc.utah.edu/sys/installdir/allinea-forge. DDT supports most available compilers, including GNU, PGI, and Intel.
In order to compile your code and run it in DDT, follow these steps:
Serial code can be debugged on personal desktops or on the interactive nodes, provided it does not generate high load that can affect responsiveness to other users.
Follow these steps:
1. Set up the environment for the appropriate compiler and DDT:module load ddt
2. Compile with -g:
For GNU compilers: g77 -g source_name.f -o executable_name
For Intel compilers: ifort -g source_name.f -o executable_name
For PGI compilers: pgf77 -g source_name.f -o executable_name
3. Run the program in Totalview:ddt ./executable_name
Parallel code can be debugged on personal desktops or on the interactive nodes using MPICH. On the interactive nodes, be considerate to other users and don't run too many processes at the same time. In case of need for more CPUs in the debugging session (our DDTlicense has 32 CPU limit), debug on the compute nodes.
To debug in parallel on interactive nodes, we recommend to use MPICH MPI implementation
due to ease of set up, although other MPI implementations should also work.
Also, since interactive nodes have limited number of processors (12 to 20) and memory
(24 to 64 GB), only parallel programs that need less processors and memory should
be debugged on interactive nodes. Still, most bugs show up on any number of processors
so we recommend debugging on the interactive nodes whenever possible since it's much
easier than debugging on compute nodes. Note that you can debug more MPI processes
than there are physical CPUcores, since debugging in general puts less load on the
CPUs and as such more than one process can be executed at a single core at the time.
The debugging process on interactive nodes is as follows:
- Load the DDT and MPICH module and check that MPICH is the default compiler
should return /uufs/chpc.utah.edu/sys/installdir/mpich/###/bin/mpicc, where ### is version number with a letter suffix. For more details on MPI implementations on HPC clusters, see CHPC's MPI Documentation.module load gcc mpich ddt
which mpicc - Compile with -g:
For GNU compilers:
g77 -g source_name.f -o executable_name
For Intel compilers:ifort -g source_name.f -o executable_name
For PGI compilers:pgf77 -g source_name.f -o executable_name - Start DDT with the program name:
ddt ./executable_name - DDT opens Run window. In this window, choose the MPI, click Details to expand the MPI menu choose number of MPI processes and pick the appropriate MPI implementation. If the code also uses OpenMP, choose the OpenMP option as well.
- Click Run button to load the program into DDT.
- Set breakpoints and click the green arrow button (Play/Continue - F9) to run the program in the debugger.
In order to debug on compute nodes, one has to run a job through the queue. This should be done only in extreme cases when the bug does not show up during a parallel MPI run on a single node or if it takes a very long time to reach the point of the bug. If the queue wait time is too long (there are no nodes available), please, contact CHPC at helpdesk@chpc.utah.edu to set up a reservation for you so that you can get the nodes you need right away.
The process is the same as in debugging on the desktop or interactive node, except that we need to submit an interactive job through the queue, and then load the DDT module and run DDT inside of the job.
DDT supports Python debugging including MPI (e.g. mpi4py), and Python libraries written in C/C++, with a few caveats.
To debug a Python code, start DDT as follows:ddt python3 %allinea_python_debug% my_python_code.py
TotalView is a full-featured, source-level, multi-process, multi-thread graphical debugger f or C, C++, Fortran, assembler, and mixed source/assembler codes.
NOTE: As of 2023, we have retired the license server that served the discontinued Totalview
license, and due to this Totalview is not available anymore. However, Totalview still
offers free licenses for students, which we encourage those who are eligible to try. We are leaving the documentation
below for legacy purposes.
NOTE: As of 2017 we have discontinued the maintenance payment for Totalview and switched to DDT. We will keep supporting the latest 2016 release of Totalview but will not update to newer versions.
More information
RogueWave, formerly TotalviewTech, manufacturer of Totalview Rogue Wave Software
Debugging with Totalview tutorial presentation (requires a video player like QuickTime, download may take some time). Video Tutorial on Totalview
Totalview is installed in /uufs/chpc.utah.edu/sys/installdir/totalview/std. This version runs on any Linux workstation with 64 bit OS. Both serial and parallel debugging (MPI, OpenMP) are supported.
Totalview supports most available compilers, including GNU, PGI, and Intel.
In order to compile your code and run it in Totalview, follow these steps:
Serial code can be debugged on personal desktops or on the interactive nodes, provided it does not generate high load that can affect responsiveness to other users.
Follow these steps:
1. Set up the environment for the appropriate compiler and Totalview. With modules,
e.g.:module load intel totalviewor with the phased out explicit environment sourcing:
source /uufs/chpc.utah.edu/sys/pkg/intel/ics/bin/compilervars.csh intel64source /uufs/chpc.utah.edu/sys/installdir/totalview/std/etc/totalview.csh
2. Compile with -g:
For GNU compilers: g77 -g source_name.f -o executable_name
For Intel compilers: ifort -g source_name.f -o executable_name
For PGI compilers: pgf77 -g source_name.f -o executable_name
3. Run the program in Totalview:totalview ./executable_name
Parallel code can be debugged on personal desktops or on the interactive nodes using MPICH. On the interactive nodes, be considerate to other users and don't run too many processes at the same time. In case of need for more CPUs in the debugging session (our Totalview license has 32 CPU limit), debug on the compute nodes.
To debug in parallel on interactive nodes, we recommend to use MPICH MPI implementation
due to ease of set up, although other MPI implementations should also work.
Also, since interactive nodes have limited number of processors (12 to 20) and memory
(24 to 64 GB), only parallel programs that need less processors and memory should
be debugged on interactive nodes. Still, most bugs show up on any number of processors
so we recommend debugging on the interactive nodes whenever possible since it's much
easier than debugging on compute nodes. Note that you can debug more MPI processes
than there are physical CPUcores, since debugging in general puts less load on the
CPUs and as such more than one process can be executed at a single core at the time.
The debugging process on interactive nodes is as follows:
- Check that MPICH is the default compiler
which mpicc
should return /uufs/chpc.utah.edu/sys/installdir/mpich/###/bin/mpicc, where ### is version number with a letter suffix. For more details on MPI implementations on HPC clusters, see CHPC's MPI Documentation. - Compile with -g:
mpif77 -g source_name.f -o executable_name - Start Totalview:
totalview &
Totalview opens New Program window. In this window, fill in the following:
* In tab Program, field Program, put in the executable program name
* In tab Parallel, field Parallel System, choose MPICH3, and for Tasks set the number of processors you want to run on
* Click OK to load the program into Totalview. After a little while, you will get a dialog box saying: Process executable.exe is a parallel job. Do you want to stop now? Click Yes, set some breakpoints and start debugging.
In order to debug on compute nodes, one has to run a job through the queue. This should be done only in extreme cases when the bug does not show up during a parallel MPI run on a single node or if it takes a very long time to reach the point of the bug. If the queue wait time is too long (there are no nodes available), please, contact CHPC at helpdesk@chpc.utah.edu to set up a reservation for you so that you can get the nodes you need right away.
- Compile with -g:
mpicc -g source_name.c -o executable_name
mpif90 -g source_name.f -o executable_name
For parallel OpenMP application using PGI compilers, include -mp to invoke OpenMP directives support, -fopenmp for GNU compilers or -openmp for Intel compilers. - Start interactive PBS session, e.g.:
sbatch -N 2 -n 24 -A account -p partition -t 2:00:00 --tty /bin/tcsh -l - Start Totalview:
totalview &
Totalview opens New Program window. In this window, fill in the following:
In tab Program, field Program, put in the executable program name
In tab Parallel, field Parallel System, choose MPICH3, and for Tasks set the number of processors you want to run on
Click OK to load the program into Totalview. After a little while, you will get a dialog box saying: Process executable.exe is a parallel job. Do you want to stop now? Click Yes, set some breakpoints and start debugging.
If you run Totalview for the first time, you may have to also specify a full path
to the TV Server, tdvsvr. To do that, modify Launch Strings dialog in Menu - Preferences to:
/uufs/chpc.utah.edu/sys/installdir/totalview/std/bin/tdvsvr
instead of just plain
tdvsvr
Click go to run the code. After a little while, you will get a dialog box saying:
Process executable.exe is a parallel job. Do you want to stop now? Click Yes, set some breakpoints and start debugging.
CHPC also owns 16 CPU license for the ReplayEngine plugin for Totalview, which enables one to step backwards in their debugging session. In order to use ReplayEngine, in the New Program dialog after starting Totalview, check the Enable ReplayEngine checkbox.
Totalview has built in memory debugging options. To enable memory debugging, in the New Program dialog after starting Totalview, check the Enable memory debugging checkbox. This option checks for array index mismatches, allocation/deallocation problems and many other common memory bugs and is useful when the program crashes for no apparent reason or when the program exits with segmentation fault error.