There is a bug in the changes we made to a part of our batch software during the downtime. The bug is affecting some users when they submit jobs to our system.

TotalView Debugger

TotalView is a symbolic debugger which supports threads, MPI, OpenMP, C/C++ and Fortran, plus mixed-language codes.  Advanced features include on-demand memory leak detection, heap allocation debugging and the Standard Template Library Viewer (STLView).  Other features like dive, a wide variety of breakpoints, the Message Queue Graph/Visualizer, powerful data analysis and control at the thread level give you the power you need to solve tough problems.  

Availability and Restrictions

TotalView is available on Ruby, Oakley, and Glenn Clusters. The versions currently available at OSC are:

Version Glenn Oakley RUBY notes
8.3.0 X(S)      
8.5.0-1 X(S)*      
8.8.0-2 X(S)      
8.9.2-1   X(CS)   Default version on Oakley prior to 09/15/2015
8.14.1-8   X* X*  
*: Current default version

NOTE:

  • X(S): standard version
  • X(CS): Both standard and CUDA versions are available
  • X: CUDA debugging included in standard module

You can use module avail totalview  to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Access

TotalView is available for use by all OSC users.

Usage

Usage on Glenn

Building your code for debugging with TotalView

You should build your code with the -g  option for symbolic debugging.  For example:

icc -g -c mycode.c

or

mpicc -g -c mympicode.c

Set-up on Glenn

To load the default standard version of Totalview module, use  module load totalview  . To select a particular software version, use   module load totalview-version . For example, use  module load totalview-8.8.0-2   to load Totalview version 8.8.0-2 on Glenn. 

Using Totalview on Glenn

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands:

module load totalview
totalview

Example code to debug can be found in $TOTALVIEW_HOME/linux-x86-64/examples .  Refer to the file README.TXT  in this directory for instructions for using these examples.

To debug an MPI program on Glenn, use the following commands in an interactive job:

module load totalview
mpiexec -tv [other mpiexec arguments] mympiprog [program arguments]

Usage on Oakley

Building your code for debugging with TotalView

You should build your code with the -g  option for symbolic debugging.  For example:

icc -g -c mycode.c

or

mpicc -g -c mympicode.c

Set-up on Oakley

To load the default standard version of Totalview module, use  module load totalview  . To select a particular software version, use   module load totalview/version . For example, use  module load totalview/8.9.2-1    to load Totalview version 8.9.2-1 on Oakley. 

Using Totalview on Oakley

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands. The -a argument is used only if you want to pass arguments to your program on the command line.

module load totalview
totalview mycode -a prog_args

Example code to debug can be found in $TOTALVIEW_HOME/linux-x86-64/examples .  Refer to the file README.TXT  in this directory for instructions for using these examples.

To debug an MPI program on Oakley, use the following commands in an interactive job:

module load totalview
mpiexec -tv [other mpiexec arguments] mympiprog [program arguments]

Totalview will stop your program at the beginning to allow you to debug. For a parallel program, the initial breakpoint is in mpiexec. Press "Go" and you will be given the option of stopping when your job goes parallel. Normally it will stop inside MPI_Init. For a parallel job running on a single node, however, it can run past that point and stop later in your program. You may need to have your program sleep for 30 seconds at the beginning.

In C:

sleep(30);

In Fortran:

call sleep(30)

Usage on Ruby

Building your code for debugging with TotalView

You should build your code with the -g  option for symbolic debugging.  For example:

icc -g -c mycode.c

or

mpicc -g -c mympicode.c

To debug an MPI program on Ruby, you must build with a debug version of mvapich2.  Load the module before building code, for example:

module load mvapich2/2.1-debug

Set-up on Ruby

To load the default standard version of Totalview module, use  module load totalview  . To select a particular software version, use   module load totalview/version . For example, use  module load totalview/8.14.1-8   to load Totalview version 8.14.1-8 on Ruby. 

Using Totalview on Ruby

To debug a serial or multithreaded program with TotalView, load the module and launch using the following commands. The -a argument is used only if you want to pass arguments to your program on the command line.

module load totalview
totalview mycode -a prog_args

Example code to debug can be found in $TOTALVIEW_HOME/linux-x86-64/examples .  Refer to the file README.TXT  in this directory for instructions for using these examples.

Use the following commands in an interactive job:

module load totalview
totalview -args mpiexec [mpiexec arguments] ./mympiprog [program arguments]

Totalview will stop your program at the beginning to allow you to debug. For a parallel program, the initial breakpoint is in mpiexec. Press "Go" and you will be given the option of stopping when your job goes parallel. Normally it will stop inside MPI_Init. For a parallel job running on a single node, however, it can run past that point and stop later in your program. You may need to have your program sleep for 30 seconds at the beginning.

In C:

sleep(30);

In Fortran:

call sleep(30)

Further Reading

  • On Oakley and Ruby, TotalView documentation in pdf format can be found in  $TOTALVIEW_HOME/doc/pdf 
  • Additional information about TotalView can be found at the Rogue Wave Software page.  
Supercomputer: 
Service: 
Fields of Science: