Sometimes the best way to get access to a piece of software on the HPC systems is to install it yourself as a "local install". This document will walk you through the OSC-recommended procedure for maintaining local installs in your home directory or project space.
Before installing your software, you should first prepare a place for it to live. We recommend the following directory structure, which you should create in the top-level of your home directory:
local |-- src | `-- tars `-- share `-- modulefiles
This structure is quite common in the UNIX world, and is also how OSC organizes the software we provide. Each directory serves a specific purpose:
local- Gathers all the files related to your local installs into one directory, rather than cluttering your home directory. Applications will be installed into this directory with the format "appname/version". This allows you to easily store multiple versions of a particular software install if necessary.
local/src- Stores the installers -- generally source directories -- for your software.
local/src/tars- Stores the compressed archives ("tarballs") of your installers. Useful if you want to reinstall later using different build options.
local/share/modulefiles- The standard place to store module files, which will allow you to dynamically add or remove locally installed applications from your environment.
You can create this structure with one command. After navigating to where you want to create the directory structure, run:
mkdir -p local/src/tars local/share/modulefiles
Finally, you need to add your local
modulefiles directory to the module system's search path. To do this, append the following line to your
.tcshrc, or other shell startup script:
module use /nfs/01/username/local/share/modulefiles
/nfs/01/username" with the full path of your home directory. (You can identify this from the command line with the command "
echo $HOME".) If you already have a .modulerc file, just add the "
module use" line to the end of it.
Now that you have your directory structure in space, you can install your software. For demonstration purposes, we will install a local copy of the Git version control system.
First, we need to get the source code onto the HPC filesystem. The easiest thing to do is find a download link, copy it, and use the
wget tool to download it on the HPC. We'll download this into
cd ~/local/src/tars wget http://git-core.googlecode.com/files/git-220.127.116.11.tar.gz
Now extract into the
src directory above. If you're working with a tar file, you can use the
-C command to specify the directory to extract to:
tar zxvf git-18.104.22.168.tar.gz -C ../
Next, we'll go into the source directory and build the program. Consult your application's documentation to determine how to specify to install into
app with the application's name and
version with the version you are installing, as demonstrated below. In this case, we'll use the
--prefix option to specify the install location.
You'll also want to specify a few variables to help make your application more compatible with our systems. We recommend specifying that you wish to use the Intel compilers and that you want to link the Intel libraries statically. This will prevent you from having to have the Intel module loaded in order to use your program. To accomplish this, add "
CC=icc CFLAGS=-static-intel" to the end of your invocation of
configure. If your application does not use
configure, you can generally still set these variables somewhere in its Makefile or build script.
With these things in mind, we can build Git using the following commands:
cd ../git-22.214.171.124 ./configure --prefix=$HOME/local/git/126.96.36.199 CC=icc CFLAGS=-static-intel make && make install
Your application should now be fully installed. However, before you can use it you will need to add the installation's directories to your path. To do this, you will need to create a module.
Modules allow you to dynamically alter your environment to define environment variables and bring executables, libraries, and other features into your shell's search paths. They are written in the Tcl language, though you do not need to be familiar with it to create a simple module.
All modules begin with the string "#%Module". After that, they contain several commands to tell the module system how to modify your environment. Some of the commonly used ones are:
prepend-path VARIABLE path- Adds
pathto the beginning of
VARIABLEis a colon-separated list of paths. Generally use to modify
setenv VARIABLE value- Sets the environment variable
set VARIABLE value- Used to set local variables to be used within the module.
You can read about all of the available commands by reading the manpage for "modulefile":
A simple module for our Git installation would be:
#%Module ## Local variables set name git set version 188.8.131.52 set root /nfs/01/username/local/$name/$version ## Environment modifications # Set basic paths prepend-path PATH $root/bin prepend-path MANPATH $root/share/man # Git includes some Python and Perl modules that may be useful prepend-path PERL5LIB $root/lib prepend-path PERL5LIB $root/lib64 prepend-path PYTHONPATH $root/lib
Any modulefile you create you should be saved into your local modulefiles directory. For maximum future-proofing, create a subdirectory within modulefiles named after your app and add one modulefile to that directory for each version of the app installed.
In the case of our Git example, you would create the directory
$HOME/local/share/modulefiles/git and create a modulefile within that directory named "
184.108.40.206". To make this module usable, you need to tell the modules utility where to look for it. You can do this by issuing the command
module use $HOME/local/share/modulefiles, in our example. This will allow you to load your app using either
module load git or
module load git/220.127.116.11. If you installed version 1.8 later on and created a modulefile for it called "
1.8", the module system would automatically load the newer version whenever you loaded
git. If you needed to go back to the older version for some reason, you can do so by specifying the version you wanted:
module load git/18.104.22.168.
module use [/path/to/modulefiles]is not persistent between sessions.
For a starting point, copy our sample modulefile from
~support/doc/modules/sample_module. This modulefile follows the recommended design patterns laid out above, and includes samples of many common module operations.
For more information about modules, be sure to read the module(1) and modulefile(4) manpages. If you have any questions about modules or local installations, feel free to contact the OSC Help Desk and firstname.lastname@example.org.