Python

Python is a high-level, multi-paradigm programming language that is both easy to learn and useful in a wide variety of applications.  Python has a large standard library as well as a large number of third-party extensions, most of which are completely free and open source. 

Availability and Restrictions

Versions

Python is available on Pitzer and Owens Clusters. The versions currently available at OSC are:

Version Owens Pitzer Ascend Cardinal Notes
2.7 X        
3.5  X        
3.6 X        
2.7-conda5.2 X X     Anaconda 5.2 distribution with Python 2.7 (conda 4.5.9 on Owens, conda 4.5.10 on Pitzer)**
3.6-conda5.2 X* X*     Anaconda 5.2 distribution with Python 3.6 (conda 4.5.9 on Owens, conda 4.5.11 on Pitzer)**

3.7-2019.10

X X     Anaconda 2019.10 distribution with Python 3.7 (conda 4.7.12)**
3.9-2022.05 X X     Anaconda 2022.05 distribution with Python 3.9 (conda 4.12.0)**
3.9     X*    
3.12       X* Anaconda 2024.06 distribution with Python 3.12.4 (conda 24.5.0)
* Current default version
** The sufix '-condaX.X' and '-20XX.XX' indicates the version of the Anaconda distribution that has been installed. These distributions encompass conda as well as various other packages. For example, python/2.7-conda5.2 has been installed with Anaconda version 5.2 but uses conda version 4.5

Some versions installed as an integrated package Anaconda

You can use module spider python to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.

Best Practices for Python Environment Management:

Utilize Miniconda3 Modules for Python Environments:Instead of relying on the default Python modules provided by OSC, leverage Miniconda3 modules for creating Python environments. Miniconda3 provides a lightweight distribution of Python and Conda, allowing for flexible environment management.
Configure Conda Channels: Before installing python packages via conda, select desired channels HOWTO: Create and Manage Python Environments based on required packages and licensing restrictions.
Maintain a Clean ~/.bashrc:It's recommended to keep your ~/.bashrc file clean and free from unnecessary scripts or Conda-related settings. This helps avoid conflicts and ensures a more predictable environment setup.
Set PYTHONNOUSERSITE before Activating Environment:Before activating a Python environment, set PYTHONNOUSERSITE=TRUE. This prevents Python from accessing and using user-installed packages located in ~/.local, ensuring a clean and isolated environment.
Deactivate Conda Environment Before Submitting Batch Jobs:Always remember to deactivate the Conda environment (conda deactivate or source deactivate) before submitting batch jobs on the HPC system. This ensures that the job runs in a clean environment without any dependencies from the active Conda environment.

Access

Python is available for use by all OSC users, but all users are required to review and accept Anaconda, Inc. Terms of Service before accessing the software.

Publisher/Vendor/Repository and License Type

Anaconda Inc., Open source and Proprietary licenses.  See Anaconda, Inc. Terms of Service for details.

Usage

Terminal

Set-up

To load the default version of Python module, use module load python . To select a particular software version, use module load python/version. For example, use module load python/3.5 to load Python version 3.5. After the module is loaded, you can run the interpreter by using the command python. To unload the Python 3.5 module, use the command module unload python/3.5 or simply module unload python

Installed Modules

We have installed a number of Python packages and tuned them for optimal performance on our systems.  When using the Anaconda distributions of python you can run conda list to view the installed packages.

NOTE:
  • Due to architecture differences between our supercomputers, we recommend NOT installing your own packages in  ~/.local. Instead, you should install them in some other directory and set $PYTHONPATH in your default environment. For more information about installing your own Python modules, please see our HOWTO.
Environments

See the HOWTO section for more information on how to create and use python environements.

Batch

When you log into owens.osc.edu or pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations (Pitzer) and Batch Limit Rules (Pitzer) for more info. 

Here is an example batch job script

#!/bin/bash
#SBATCH --account <your_project_id>
#SBATCH --job-name Python_ExampleJob
#SBATCH --nodes=1 
#SBATCH --time=00:01:00

module load python/3.9-2022.05

cp example.py $TMPDIR
cd $TMPDIR

python example.py

cp -p * $SLURM_SUBMIT_DIR
    

Utilizing Python Environments Within Batch Job:

Important: When utilizing a python environment make sure to deactivate the environment before submitting the script or include  source deactivate in the batch script before activating the environment.
Here is an example batch job script involving conda environment:
#!/bin/bash
#SBATCH --account <your_project_id>
#SBATCH --job-name Python_ExampleJob
#SBATCH --nodes=1
#SBATCH --time=00:01:00

# run to following to ensure local environment does not effect the batch job in unexpected ways

source deactivate # deactivate copy of local python environment if job submitted from within environment
module reset      # reset any loaded modules

module load python/3.9-2022.05 # load python
export PYTHONNOUSERSITE=True  #to avoid local python packages

source activate MY_ENV  # activate conda environment 


# Rest of script below

cp example.py $TMPDIR

cd $TMPDIR

python example.py

cp -p * $SLURM_SUBMIT_DIR

HOW-TOs

Use Jupyter on OnDemand

OnDemand allows for use of the Jupyter interactive app. Please refer to the following page for more details:

Manage your Python packages

We highly recommend creating a local environment using Miniconda3 modules to manage Python packages for your production and research tasks. Please refer to the following how-to pages for more details:

Install packages for deep/machine learning

Advanced topics

 

Known Issues

Incorrect MPI launcher and compiler wrappers with Conda environments

Updated: March 2020
Versions Affected: Python 2.7, 3.6 & Conda 5.2
Users may encounter under-performing MPI jobs or failures of compiling MPI applications if you are using Conda from system. We found pre-installed mpich2 package in some Conda environments overrides default MPI path. The affected Conda packages are python/2.7-conda5.2 and python/3.6-conda5.2. If users experience these issues, please re-load MPI module, e.g. module load mvapich2 after setting up your Conda environment.
 

Compatibility Issues with NumPy 2.0

 

The newly released version of NumPy 2.0 includes substantial internal changes, including migrating code from C to C++. These modifications have led to significant issues with backwards compatibility, resulting in numerous breaking changes to both the Python and C APIs. As a consequence, packages built against NumPy 1.xx may encounter ImportError messages. To ensure compatibility, these packages must be rebuilt against NumPy 2.0.

Recommendation for Addressing the Issue:

  1. Follow the Migration Guide: Refer to the NumPy 2.0 Migration Guide for detailed instructions.

  2. Update Packages: Ensure all packages are updated to their latest versions.

  3. Contact Developers: Reach out to package developers for updates or compatibility information.

  4. Create a Project-Specific Environment: Set up a dedicated Python environment for your project to manage package versions effectively. Refer to the OSC documentation for guidance on using the Conda package manager.

  5. Separate Environments for Compatibility: Maintain separate Python environments for packages that are compatible with NumPy 1.x and NumPy 2.x.

Further reading

Extensive documentation of the Python programming language and software downloads can be found at the Official Python Website.  

See Also

Service: 
Fields of Science: