conda/analysis3 Python Environment

Danger

All users are advised to update their workflows to replace the hh5 conda/analysis3 environment with the xp65 conda/analysis3 environment as soon as possible to ensure continued support and access to the latest features. The hh5 conda/analysis3 environment is no longer actively maintained and, after 30 May 2025, may be taken down without prior notice.

ACCESS-NRI now supports and maintains the conda/analysis3 Python environment, housed within the xp65 NCI project. This environment includes Python libraries commonly used for climate data processing and analysis, allowing users to run workflows on Gadi without having to manage package installations themselves. This is the continuation of the environments formerly maintained by CLEX within the hh5 NCI project.

How to use the xp65 conda/analysis3 environment

There are 3 main ways to use the conda/analysis3 environment:

Command line and shell scripts
PBS job
ARE

Tip

If you only want to update your workflows from the previous hh5 conda/analysis3 environment, refer to Update your workflows below.

Manually load the environment

For each new session within a Gadi login node, ARE VDI terminal, or in a shell script, run:

module use /g/data/xp65/public/modules
module load conda/analysis3

Tip

If you have previously added commands to load the hh5 conda/analysis3 environment within your .bashrc or .bash_profile file, we strongly recommend completely removing those lines, as programmatically loading environments that way might lead to unexpected interference with other processes on Gadi.

A safer approach is to define a Bash alias or shell function that runs your desired series of commands. You can then manually invoke the alias whenever needed.
For example, you can add the following line to your .bashrc or .bash_profile file:

alias analysis3='module use /g/data/xp65/public/modules && module load conda/analysis3'

Then, within any session, you can simply run analysis3 to load the environment.

Use the environment within a PBS job

In a PBS job submission script (including usage within rose/cylc workflows), in addition to adding the module use and module load lines above, add gdata/xp65 to your storage flag:

#PBS -l storage=gdata/xp65

Tip

Multiple storage projects are separated by a plus (+):

#PBS -l storage=scratch/tm70+gdata/xp65

Use the environment within ARE

When launching an ARE JupyterLab instance:

Under Storage, add gdata/xp65.
If you have other storage locations, use a plus sign (+) (e.g., +gdata/xp65).
Under Advanced options:
- In "Module directories", add /g/data/xp65/public/modules/
- In "Modules", add conda/analysis3

When launching an ARE VDI instance:

Under Storage, add gdata/xp65.
If you have other storage locations, use a plus sign (+) (e.g., +gdata/xp65).
Under PBS flags, add -v SINGULARITY_OVERLAYIMAGE=/g/data/xp65/public/apps/med_conda/envs/$(readlink /g/data/xp65/public/apps/med_conda/envs/analysis3).sqsh

Warning

This will only work if you load the latest analysis3 environment using module load conda/analysis3.
If you want to load a specific version, the SINGULARITY_OVERLAYIMAGE env variable will need to specify the exact version in the format: /g/data/xp65/public/apps/med_conda/envs/analysis3-YY-MM.sqsh.

How to switch from the hh5 to the xp65 conda/analysis3 environment

If you have workflows that use the hh5 conda/analysis3 environment, follow the steps below to update them to use the xp65 conda/analysis3 environment instead:

If you have not already, request membership to the xp65 project
Replace all instances of hh5 with xp65
Make sure you are loading a correct version
The xp65 conda/analysis3 environment versioning might differ from the old hh5 conda/analysis3 one. Therefore, if you were loading a specific version (e.g., conda/analysis3-23.10), make sure to specify a valid version for the xp65 conda/analysis3 environment.

Advanced usage

Two Series of Environments: conda/analysis3 and conda/analysis3-edge

There are two parallel series of environments:

conda/analysis3
A stable environment designed for long-term usability with well-tested, reliable package versions.
conda/analysis3-edge
A cutting-edge environment with the latest available packages, ideal for those needing the most up-to-date software and features.

These series offer flexibility: use conda/analysis3 for stability, or conda/analysis3-edge for the latest updates.

Load a specific environment version

The xp65 conda/analysis3 environment follows the versioning format conda/analysis3-YY-MM, where YY represents the last two digits of the release year and MM indicates the release month.

Command line and PBS

When manually loading the environment or within a PBS job, you can load a particular version by explicitly specifying its release year and month (e.g., to load the April 2025 environment, specify conda/analysis3-25.04).
If you do not specify a version (e.g. using conda/analysis3 rather than conda/analysis3-YY.MM) the latest (i.e., previous month's) frozen environment version will be loaded. We encourage those who want the newest features to help us test the current version by explicitly loading the current month’s environment.
module use /g/data/xp65/public/modules module load conda/analysis3 Loading conda/analysis3-25.06 Loading requirement: singularity

Note

For rose/cylc workflows, it is not recommended to specify a particular version of the environment (i.e. use conda/analysis3 and not conda/analysis3-YY.MM).

ARE Jupyterlab

When launching an ARE Jupyterlab session, you only need to include conda/analysis3. If you would like to specify a particular environment version, you can do so for each notebook by switching kernels inside the ARE instance.

Tip

The same version naming structure can be applied to the conda/analysis3-edge environment.

Acknowledgements

These environments, developed and maintained by the CLEX CMS team, have proven very valuable to the community over the years. We commend and thank the CMS team for implementing this very successful service and supporting it for the whole community.