HOWTO: Tune VASP Memory Usage

This article discusses memory tuning strategies for VASP.

Data Distribution

Typically the first approach for memory sensitive VASP issues is to tweak the data distribution (via NCORE or NPAR). The information at https://www.vasp.at/wiki/index.php/NPAR covers a variety of machines. OSC has fast communications via Infiniband. 

Performance and memory consumption are dependent on the simulation model.  So we recommend a series of benchmarks varying the number of nodes and NCORE. The recommended initial value for NCORE is the processor count per node which is the ntasks-per-node value in Slurm (the ppn value in PBS). Of course, if this benchmarking is intractable then one must reexamine the model. For general points see:  https://www.vasp.at/wiki/index.php/Memory_requirements and https://www.vasp.at/wiki/index.php/Not_enough_memory  And of course one should start small and incrementally improve or scale up one's model.

Rationalization

Using the key parameters with respect to memory scaling listed at the VASP memory requirements page one can rationalize VASP memory usage.  The general approach is to study working calculations and then apply that understanding to scaled up or failing calculations.  This might help one identify if a calculation is close to a node's memory limit and happens to cross over the limit for reasons that might be out of ones control, in which case one might need to switch to higher memory nodes.

Here is an example of rationalizing memory consumption.  Extract from a simulation output the key parameters:

Dimension of arrays:
k-points NKPTS = 18 k-points in BZ NKDIM = 18 number of bands NBANDS= 1344
total plane-waves NPLWV = 752640
...
dimension x,y,z NGXF= 160 NGYF= 168 NGZF= 224
support grid NGXF= 320 NGYF= 336 NGZF= 448

This yields 273 GB of memory, NKDIM*NBANDS*NPLWV*16 + 4*(NGXF/2+1)*NGYF*NGZF*16, according to
https://www.vasp.at/wiki/index.php/Memory_requirements

This estimate should be compared to actual memory reports.  See for example XDModD and grafana.  Note that most application software has an overhead in the ballpack of ten to twenty percent.  In addition, disk caching can consume significant memory.  Thus, one must adjust the memory estimate upward.  It can then be comapred to the available memory per cluster and per cluster node type.

Miscellaneous

  • OSC sets the default resource limits for shells, except for core dump file size, to unlimited; see the limit/ulimit/unlimit commands depending on your shell.
  • In the INCAR input file NWRITE=3 is for verbose output and NWRITE=4 is for debugging output.
  • OSC does not have a VASP license and our staff has limited experience with it.  So investigate alternate forms of help:  ask within your research group and  post on the VASP mailing list.
  • Valgrind is a tool that can be used for many types of debugging including looking for memory corruptions and leaks. However, it slows down your code a very sizeable amount. This might not be feasible for HPC codes
  • ASAN (address sanitizer) is another tool that can be used for memory debugging. It is less featureful than Valgrind, but runs much quicker, and so will likely work with your HPC code.

 

Supercomputer: 
Owens
Pitzer
Service: 
Training