Supercomputing FAQ

General Questions

Account Questions

Disk Storage Questions

Email Questions

Linux Questions

SSH Questions

Batch Processing Questions

Compiling System Questions

Parallel Processing Questions

Libraries/Software Questions

Performance Analysis Questions

Other Common Problems

General Questions

Who can get an account?

Anyone can have an account with OSC, but you need access to a project to utilize our resources. If an eligible principal investigator has a current project, he/she can add the user through client protal MyOSC. Authorized users do not have to be located in Ohio or at the same institution. 

See our webpage for more information: https://www.osc.edu/supercomputing/support/account

Where should a new OSC user begin?

Once you are able to connect to our HPC systems, you should start familiarizing yourself with the software and services available from the OSC, including:

Do I have to pay for supercomputer use?

It depends on the type of client and your rate of consumption. Please click here for more information.

How many supercomputers does OSC have?  Which one should I use?

OSC currently has three HPC clusters: Pitzer Cluster, a 29,664 core Dell cluster with Intel Xeon proccessors, Owens Cluster, a 23,500+ core Dell cluster with Intel Xeon processors, and Ascend Cluster with 2,304 core Dell cluster devoted to intensive GPU processing. New users have access to Pitzer and Owens clusters. To learn more,click here.

How do I cite OSC in my publications?

Any publication of any material, whether copyrighted or not, based on or developed with OSC services, should cite the use of OSC, and the use of the specific services (where applicable). For more information about citing OSC, please visit www.osc.edu/citation.

How do I submit my publications and funding information to OSC?

You can add these to your profile in MyOSC. You can then associate them with OSC project(s). 

See our website for more information: https://www.osc.edu/supercomputing/portals/client_portal/manage_profile_information 

Can I rceive a letter of support from OSC when I apply for outside funding?

OSC has a standard letter of support that you can include (electronically or in hard copy) with a proposal for outside funding. This letter does not replace the budget process. To receive the letter of support, please send your request to oschelp@osc.edu. You should provide the following information: name and address of the person/organization to whom the letter should be addressed; name(s) of the principal investigator(s) and the institution(s); title of the proposal; number of years of proposed project; budget requested per year. Please allow at least two working days to process your request.

Hardware information about the systems is available at http://www.osc.edu/supercomputing/hardware

How do I register for a workshop?

For a complete schedule of current training offerings, please visit the OSC Training Schedule.  To register or for more information, please email oschelp@osc.edu.  

Where can I find documentation?

For documentation specific to software applications, see Software.  For other available hardware, see Supercomputers.

My question isn't answered here. Whom can I ask for help?

Contact the OSC Help Desk. Our regular business hours are Monday - Friday, 9am - 5pm. More information on the OSC supercomputing help desk can be found on our Support Services page.

Something seems to be wrong with the OSC systems.  Should I contact the help desk?

Information will be coming soon for guidelines on reporting possible system problems.

Where can I find logos for my presentations, posters, etc.?

Please see our citation webpage.

Account Questions

What are projects and accounts?

An eligible principal investigator heads a project. Under a project, authorized users have accounts with credentials that permit users to gain access to the HPC systems. A principal investigator can have more than one project. 

How do I get/renew an account?

For information concerning accounts (i.e., how to apply, who can apply, etc.), see Accounts.

I'm a faculty member.  How do I get accounts for my students?

If an eligible principal investigator is new to OSC, he/she can create a new project. If an eligible principal investigator has a current project, he/she can add the user through client protal MyOSC. Authorized users do not have to be located in Ohio or at the same institution.

I'm continuing the research of a student who graduated.  Can I use his/her account?

Please have your PI send an email to oschelp@osc.edu for further discussions. 

I'm working closely with another student.  Can we share an account?

No. Each person using the OSC systems must have his/her own account. Sharing files is possible, even with separate accounts. 

How do I change my password?

You can change your password through the MyOSC portal. Log in at MyOSC, and click your name in the upper right hand corner to secure a dropdown menu. Select the "change password" item. Please note that your password has certain requirements; these are specified on the "change password" portal. You may need to wait up to 20 minutes to be able to login with the new password. For security purposes, please note that our password change policy requires a password change every 180 days.

If your password has expired, you can update by following the "Forgot your password?" link at MyOSC login page.

I want to use csh instead of bash.  How do I change the default shell?

You can change your default shell through the MyOSC portal. Log in at MyOSC, and use the "Unix Shell" drop-down menu in the HPC User Profile box to change your shell. You will need to log off the HPC system and log back on before the change goes into effect. Please note, that it will take about a few minutes for the changes to be applied.

How do I find my project budget balance?

To see usage and balance information from any system, refer to the OSCusage page.

NOTE: Accounting is updated once a day, so the account balance is for the previous day.

How do I get more resources?

To request additional use of our resources, the principal investigator will need to change the budget for their project. Please see the creating budgets and projects page.

How much will my project be charged for supercomputer usage?

If the project is associated with an Ohio academic institution, see the academic fee structure page for pricing.

If the project is NOT associated with an Ohio academic institution, contact OSC Sales for information on pricing.

See Job and storage charging for how OSC calculates charges.

Disk Storage Questions

What is my disk quota?

Each user has a quota of 500 gigabytes (GB) of storage and 1,000,000 files. You may also have access to a project directory with a separate quota.  See Available File Systems for more information.

How can I determine the total disk space used by my account?

Your quota and disk usage are displayed every time you log in. You have limits on both the amount of space you use and the number of files you have. There are separate quotas for your home directory and any project directories you have access to.

Note: The quota information displayed at login is updated twice a day, so the information may not reflect the curent usage.

You may display your home directory quota information with quota -s .

How do I get more disk space?

Your home directory quota cannot be increased. You should consider deleting, transferring, and/or compressing your files to reduce your usage. 

A PI may request project space to be shared by all users on a project. Estimate the amount of disk space that you will need and the duration that you will need it. Send requests to oschelp@osc.edu.

How can I find my largest directories?

To reveal the directories in your account that are taking up the most disk space you can use the du , sort and tail commands. For example, to display the ten largest directories, change to your home directory and then run the command:

du . | sort -n | tail -n 10

Why do I receive "no space left" error when writing data to my home directory?

If you receive the error "No space left on device" when you try to write data to your home directory, it indicates the disk is full. First, check your home directory quota. Each user has 500 GB quota of storage and the quota information is shown when you login to our systems. If your disk quota is full, consider reducing your disk space usage. If your disk quota isn't full (usage less than 500GB), it is very likely that your disk is filled up with 'snapshot' files, which are invisible to users and used to track fine-grained changes to your files for recovering lost/deleted files. In this case, please contact OSC Help for further assistance. To avoid this situation in future, consider running jobs that do a lot of disk I/O in the temporary filesystem ($TMPDIR or $PFSDIR) and copy the final output back at the end of the run. See Available File Systemsfor more information.

How can I use tar and gzip to aggregate and compress my files?

The commands tar and gzip can be used together to produce compressed file archives representing entire directory structures. These allow convenient packaging of entire directory contents. For example, to package a directory structure rooted at src/ use

tar -czvf src.tar.gz src/

This archive can then be unpackaged using

tar -xzvf src.tar.gz

where the resulting directory/file structure is identical to what it was initially.

The programs zip , bzip2 and compress can also be used to create compressed file archives. See the man pages on these programs for more details.

Tar is taking too long.  Is there a way to compress quicker?

If using tar with the options zcvf is taking too long you can instead use pigz  in conjunction with tar.   pigz does gzip compression while taking advantage of multiple cores.

tar cvf - paths-to-archive | pigz > archive.tgz

pigz defaults to using eight cores, but you can have it use more or less with the -p argument.

tar cvf - paths-to-archive | pigz -n 4 > archive.tgz
Due to the parallel nature of pigz, if you are using it on a login node you should limit it to using 2 cores.  If you would like to use more cores you need to submit either an interactive or batch job to the queue and do the compression from within the job.

Note: pigz does not significantly improve decompression time.  

Email Questions

How do I change the email address OSC uses to contact me?

Please update your email on MyOSC, or send your new contact information to oschelp@osc.edu. 

I got an automated email from OSC.  Where can I get more information about it?

See the Knowledge Base.

Linux Questions

What is Linux?

Linux is an open-source operating system that is similar to UNIX. It is widely used in High Performance Computing.

How can I get started using Linux?

See the Unix Basics tutorial for more information.  There are also many tutorials available on the web.

SSH Questions

What is SSH?

Secure Shell (SSH) is a program to log into another computer over a network, to execute commands in a remote machine, and to move files from one machine to another. It provides strong authentication and secure communications over insecure channels. SSH provides secure X connections and secure forwarding of arbitrary TCP connections.

How does SSH work?

SSH works by the exchange and verification of information, using public and private keys, to identify hosts and users. The  ssh-keygen command creates a directory ~/.ssh and files that contain your authentication information. The public key is stored in ~/.ssh/id_rsa.pub and the private key is stored in  ~/.ssh/id_rsa. Share only your public key. Never share your private key. To further protect your private key you should enter a passphrase to encrypt the key when it is stored in the file system. This will prevent people from using it even if they gain access to your files. 

One other important file is ~/.ssh/authorized_keys. Append your public keys to the authorized_keys file and keep the same copy of it on each system where you will make ssh connections. 

In addition, on Owens the default SSH client config enables hashing of a user’s known_hosts file.  So if SSH is used on Owens the remote system’s SSH key is added to ~/.ssh/known_hosts in a hashed format which can’t be unhashed.  If the remote server’s SSH key changes, special steps must be taken to remove the SSH key entry:

ssh-keygen -R <hostname>

Can I connect without using an SSH client?

The OSC OnDemand portal allows you to connect to our systems using your web browser, without having to install any software. You get a login shell and also the ability to transfer files.  

How can I upload or download files?

Most file transfers are done using sftp (SSH File Transfer Protocol) or scp (Secure CoPy). These utilities are usually provided on Linux/UNIX and Mac platforms. Windows users should read the next section, "Where can I find SSH and SFTP clients".

Where can I find SSH and SFTP clients?

There are many SSH and SFTP clients available, both commercial and free.  See Getting Connected for some suggestions.  

How do I run a graphical application in an SSH session?

Graphics are handled using the X11 protocol. You’ll need to run an X display server on your local system and also set your SSH client to forward (or "tunnel") X11 connections. On most UNIX and Linux systems, the X server will probably be running already. On a Mac or Windows system, there are several choices available, both commercial and free. See our guide to Getting Connected for some suggestions.

Why do I get "connection refused" when trying to connect to a cluster?

OSC temporarily blacklists some IP addresses when multiple failed logins occur. If you are connecting from behind a NAT gateway, as is commonly used for public or campus wireless networks, and get a "connection refused" message it is likely that someone recently tried to connect multiple times and failed when connected to the same network you are on. Please contact OSC Help with your public IP address and the cluster you attempted to connect to and we will remove your IP from the blacklist. You can learn your public IP by searching for "what is my IP address" in Google.

Batch Processing Questions

What is a batch request?

On all OSC systems, batch processing is managed by the Simple Linux Utility for Resource Management system (Slurm). Slurm batch requests (jobs) are shell scripts that contain the same set of commands that you enter interactively. These requests may also include options for the batch system that provide timing, memory, and processor information. For more information, see our guide to Batch Processing at OSC.  

How do I submit, check the status, and/or delete a batch job?

Slurm uses sbatch  to submit, squeue to check the status, and scancel  to delete a batch request. For more information, see our Batch-Related Command Summary. 

Can I be notified by email when my batch job starts or ends?

Yes.  See the --mail-type  option in our Slurm docoumentation. If you are submitting a large number of jobs, this may not be a good idea.

Why won't my job run?

There are numerous reasons why a job might not run even though there appear to be processors and/or memory available. These include:

  • Your account may be at or near the job count or processor count limit for an individual user.
  • Your group/project may be at or near the job count or processor count limit for a group.
  • The scheduler may be trying to free enough processors to run a large parallel job.
  •  Your job may need to run longer than the time left until the start of a scheduled downtime.
  • You may have requested a scarce resource or node type, either inadvertently or by design.

See our Scheduling Policies and Limits for more information. 

How can I retrieve files from unexpectedly terminated jobs?

A batch job that terminates before the script is completed can still copy files from $TMPDIR to the user's home directory via the use of signals handling. In the batch script, there should be an additional sbatch option added for --signals. See Signal handling in job scripts for details.

If a command in a batch script is killed for excessive memory usage (see Out-of-Memory (OOM) or Excessive Memory Usage for details) then the handler may not be able to fully execute it's commands. However, normal shell scripting can handle this situation:  the exit status of a command that may possibly cause an OOM can be checked and appropriate action taken.  Here is a Bourne shell example:

bla_bla_big_memory_using_command_that_may_cause_an_OOM
if [ $? -ne 0 ]; then
cd $SLURM_SUBMIT_DIR;mkdir $SLURM_JOB_ID;cp -R $TMPDIR/* $SLURM_JOB_ID
exit
fi

Finally, if a node your job is running on crashes then the commands in the signal handler may not be executed. It may be possible to recover your files from batch-managed directories in this case. Contact OSC Help for assistance.

How can I delete all of my jobs on a cluster?

To delete all your jobs on one of the clusters, including those currently running, queued, and in hold, login to the cluster and run the command:

scancel -u <username>

How can I determine the number of cores in use by me or my group?

# current jobs queued/running and cpus requested
squeue --cluster=all --account=<proj-code> --Format=jobid,partition,name,timeLeft,timeLimit,numCPUS
# or for a user
squeue --cluster=all -u <username> --Format=jobid,partition,name,timeLeft,timeLimit,numCPUS

How to request GPU nodes for visualization?

By default, we don't start an X server on gpu nodes because it impacts computational performance. Add vis  in your GPU request such that the batch system uses the GPUs for visualization. For example, on Owens, it should be

--nodes=1 --ntasks-per-node=28 --gpus-per-node=1 --gres=vis

Compiling System Questions

What languages are available?

Fortran, C, and C++ are available on all OSC systems. The commands used to invoke the compilers and/or loaders vary from system to system. For more information, see our Compilation Guide.

What compiler (vendor) do you recommend?

We have Intel, PGI, and gnu compilers available on all systems. Each compiler vendor supports some options that the other doesn’t, so the choice depends on your individual needs.For more information, see our Compilation Guide.

Will software built for one system run on another system?

Most serial code built on one system will run on another system, although it may run more efficiently if it is built and run on the same system. Parallel (MPI) code typically must be built on the system where it will run.

What is the difference between installing software on one's local computer and on an OSC cluster?

One major difference is that OSC users cannot install software system wide using package managers. In general, users installing software in their home directories will follow the configure/build/test paradigm that is common on Unix-like operating systems.For more information, see our HOWTO: Locally Installing Software on an OSC cluster.

What is this build error: "... relocation truncated to fit ..."?

OSC users installing software on a cluster occasionally report this error. It is related to memory addressing and is usually fixed by cleaning the current build and rebuilding with the compiler option "-mcmodel=medium". For more details, see the man page for the compiler.

Parallel Processing Questions

What is parallel processing?

Parallel processing is the simultaneous use of more than one computer (or processor) to solve a problem. There are many different kinds of parallel computers. They are distinguished by the kind of interconnection between processors or nodes (groups of processors) and between processors and memory.

What parallel processing environments are available?

On most systems, both shared-memory and distributed-memory parallel programming models can be used. Versions of OpenMP (for multithreading or shared-memory usage) and MPI (for message-passing or distributed-memory usage) are available.  A summary of parallel environments will be coming soon.

What is a core?

A core is a processor. When a single chip contains multiple processors, they are called cores.

I'm not seeing the performance I expected.  How can I be sure my code is running in parallel?

We are currently working on a guide for this. Please contact OSC Help for assistance.

Libraries/Software Questions

What software applications are available?

See the Software section for more information.

Do you have a newer version of (name your favorite software)?

Check the Software section to see what versions are installed. You can also check the installed modules using the module spider  or module avail  command.

How do I get authorized to use a particular software application?

Please contact OSC Help for assistance.

What math routines are available?  Do you have ATLAS and LAPACK?

See the Software section for information on third-party math libraries (e.g., MKL, ACML, fftw, scalapack, etc). MKL and ACML are highly optimized libraries that include the BLAS and LAPACK plus some other math routines. 

Do you have NumPy/SciPy?

The NumPy and SciPy modules are installed with the python software. See the Python software page.

OSC does not have a particular software package I would like to use. How can I request it?

Download the Request for Software Form. Once it is complete, attach the form to an e-mail to oschelp@osc.edu. 

You may install open source software yourself in your home directory. If you have your own license for commercial software, contact the OSC Help desk.

I have a software package that must be installed as root.  What should I do?

Most packages have a (poorly documented) option to install under a normal user account. Contact the OSC Help desk if you need assistance. We generally do not install user software as root.

What are modules?

Modules are used to manage the environment variable settings associated with software packages in a shell-independent way. On OSC's systems, you will by default have modules in your environment for PBS, MPI, compilers, and a few other pieces of software. For information on using the module system, see our guide to Batch Processing at OSC.

Performance Analysis Questions

What are MFLOPS/GFLOPS/TFLOPS/PFLOPS?

MegaFLOPS/GigaFLOPS/TeraFLOPS/PetaFLOPS are millions/billions/trillions/quadrillions of FLoating-point Operations (calculations) Per Second.

How do I find out about my code's performance?

A number of performance analysis tools are available on OSC systems. Some are general to all systems and others are specific to a particular system. See our performance analysis guide for more info.

How can I optimize my code?

There are several ways to optimize code. Key areas to consider are CPU optimization, I/O optimization, memory optimization, and parallel optimization. See our optimization strategy guide for more info.

Other Common Problems

What does "CPU time limit exceeded" mean?

Programs run on the login nodes are subject to strict CPU time limits. To run an application that takes more time, you need to create a batch request. Your batch request should include an appropriate estimate for the amount of time that your application will need. See our guide to Batch Processing at OSC for more information.

My program or file transfer died for no reason after 20 minutes.  What happened?

Programs run on the login nodes are subject to strict CPU time limits. Because file transfers use encryption, you may hit this limit when transferring a large file. To run longer programs, use the batch system. To transfer larger files, connect to sftp.osc.edu instead of to a login node.

Why did my program die with a segmentation fault, address error, or signal 11?

This is most commonly caused by trying to access an array beyond its bounds -- for example, trying to access element 15 of an array with only 10 elements. Unallocated arrays and invalid pointers are other causes. You may wish to debug your program using one of the available tools such as the TotalView Debugger.

I created a batch script in a text editor on a Windows or Mac system, but when I submit it on an OSC system, almost every line in the script gives an error. Why is that?

Windows and Mac have different end-of-line conventions for text files than UNIX and Linux systems do, and most UNIX shells (including the ones interpreting your batch script) don't like seeing the extra character that Windows appends to each line or the alternate character used by Mac. You can use the following commands on the Linux system to convert a text file from Windows or Mac format to UNIX format:

dos2unix myfile.txt 

mac2unix myfile.txt

I copied my output file to a Windows system, but it doesn't display correctly.  How can I fix it?

A text file created on Linux/UNIX will usually display correctly in Wordpad but not in Notepad. You can use the following command on the Linux system to convert a text file from UNIX format to Windows format:

unix2dos myfile.txt

What IP ranges do I need to allow in my firewall to use OSC services?

See our knowledge base article on the topic.

Supercomputer: 
Owens
Pitzer