Python is a high-level, multi-paradigm programming language that is both easy to learn and useful in a wide variety of applications. Python has a large standard library as well as a large number of third-party extensions, most of which are completely free and open source.
Availability and Restrictions
Versions
Python is available on Pitzer and Owens Clusters. The versions currently available at OSC are:
Version | Owens | Pitzer | Ascend | Cardinal | Notes |
---|---|---|---|---|---|
2.7 | X | ||||
3.5 | X | ||||
3.6 | X | ||||
2.7-conda5.2 | X | X | Anaconda 5.2 distribution with Python 2.7 (conda 4.5.9 on Owens, conda 4.5.10 on Pitzer)** | ||
3.6-conda5.2 | X* | X* | Anaconda 5.2 distribution with Python 3.6 (conda 4.5.9 on Owens, conda 4.5.11 on Pitzer)** | ||
3.7-2019.10 |
X | X | Anaconda 2019.10 distribution with Python 3.7 (conda 4.7.12)** | ||
3.9-2022.05 | X | X | Anaconda 2022.05 distribution with Python 3.9 (conda 4.12.0)** | ||
3.9 | X* | ||||
3.12 | X* | Anaconda 2024.06 distribution with Python 3.12.4 (conda 24.5.0) |
Some versions installed as an integrated package Anaconda.
You can use module spider python
to view available modules for a given machine. Feel free to contact OSC Help if you need other versions for your work.
Best Practices for Python Environment Management:
~/.bashrc
file clean and free from unnecessary scripts or Conda-related settings. This helps avoid conflicts and ensures a more predictable environment setup.PYTHONNOUSERSITE=TRUE
. This prevents Python from accessing and using user-installed packages located in ~/.local
, ensuring a clean and isolated environment.conda deactivate
or source deactivate
) before submitting batch jobs on the HPC system. This ensures that the job runs in a clean environment without any dependencies from the active Conda environment.Access
Python is available for use by all OSC users, but all users are required to review and accept Anaconda, Inc. Terms of Service before accessing the software.
Publisher/Vendor/Repository and License Type
Anaconda Inc., Open source and Proprietary licenses. See Anaconda, Inc. Terms of Service for details.
Usage
Terminal
Set-up
To load the default version of Python module, use module load python
. To select a particular software version, use module load python/version
. For example, use module load python/3.5
to load Python version 3.5. After the module is loaded, you can run the interpreter by using the command python
. To unload the Python 3.5 module, use the command module unload python/3.5
or simply module unload python
.
Installed Modules
We have installed a number of Python packages and tuned them for optimal performance on our systems. When using the Anaconda distributions of python you can run conda list
to view the installed packages.
- Due to architecture differences between our supercomputers, we recommend NOT installing your own packages in
~/.local
. Instead, you should install them in some other directory and set$PYTHONPATH
in your default environment. For more information about installing your own Python modules, please see our HOWTO.
Environments
See the HOWTO section for more information on how to create and use python environements.
Batch
When you log into owens.osc.edu or pitzer.osc.edu you are actually logged into a linux box referred to as the login node. To gain access to the mutiple processors in the computing environment, you must submit your job to the batch system for execution. Batch jobs can request mutiple nodes/cores and compute time up to the limits of the OSC systems. Refer to Queues and Reservations (Pitzer) and Batch Limit Rules (Pitzer) for more info.
Here is an example batch job script
#!/bin/bash #SBATCH --account <your_project_id> #SBATCH --job-name Python_ExampleJob #SBATCH --nodes=1 #SBATCH --time=00:01:00 module load python/3.9-2022.05 cp example.py $TMPDIR cd $TMPDIR python example.py cp -p * $SLURM_SUBMIT_DIR
Utilizing Python Environments Within Batch Job:
source deactivate
in the batch script before activating the environment.#!/bin/bash #SBATCH --account <your_project_id> #SBATCH --job-name Python_ExampleJob #SBATCH --nodes=1 #SBATCH --time=00:01:00 # run to following to ensure local environment does not effect the batch job in unexpected ways source deactivate # deactivate copy of local python environment if job submitted from within environment module reset # reset any loaded modules module load python/3.9-2022.05 # load python export PYTHONNOUSERSITE=True #to avoid local python packages source activate MY_ENV # activate conda environment # Rest of script below cp example.py $TMPDIR cd $TMPDIR python example.py cp -p * $SLURM_SUBMIT_DIR
HOW-TOs
Use Jupyter on OnDemand
OnDemand allows for use of the Jupyter interactive app. Please refer to the following page for more details:
Manage your Python packages
We highly recommend creating a local environment using Miniconda3 modules to manage Python packages for your production and research tasks. Please refer to the following how-to pages for more details:
- Create a Local Python Environment
- Install Python Packages from source
- Use Local Python Environment with Jupyter
Install packages for deep/machine learning
Advanced topics
- Use GPU for Python
- Run Python in Parallel
- Python for Classroom (Intended for PI accounts, not for general users)
Known Issues
Incorrect MPI launcher and compiler wrappers with Conda environments
Versions Affected: Python 2.7, 3.6 & Conda 5.2
python/2.7-conda5.2
and python/3.6-conda5.2
. If users experience these issues, please re-load MPI module, e.g. module load mvapich2
after setting up your Conda environment.Compatibility Issues with NumPy 2.0
The newly released version of NumPy 2.0 includes substantial internal changes, including migrating code from C to C++. These modifications have led to significant issues with backwards compatibility, resulting in numerous breaking changes to both the Python and C APIs. As a consequence, packages built against NumPy 1.xx may encounter ImportError messages. To ensure compatibility, these packages must be rebuilt against NumPy 2.0.
Recommendation for Addressing the Issue:
-
Follow the Migration Guide: Refer to the NumPy 2.0 Migration Guide for detailed instructions.
-
Update Packages: Ensure all packages are updated to their latest versions.
-
Contact Developers: Reach out to package developers for updates or compatibility information.
-
Create a Project-Specific Environment: Set up a dedicated Python environment for your project to manage package versions effectively. Refer to the OSC documentation for guidance on using the Conda package manager.
-
Separate Environments for Compatibility: Maintain separate Python environments for packages that are compatible with NumPy 1.x and NumPy 2.x.
Further reading
Extensive documentation of the Python programming language and software downloads can be found at the Official Python Website.