Run models using Rose/Cylc
Warning
ACCESS models configurations that run with Rose/Cylc currently use Cylc7, with intentions to upgrade to Cylc8. The upgrade is expected to change some aspects of the workflow described on this page. Updated information about the Cylc8 workflow will be provided once a model configuration using this version becomes available.
About
The Rose/Cylc workflow management tool consists of two components:
- The Cylc (pronounced ‘silk’) task engine, developed by the New Zealand National Institute of Water and Atmospheric Research (NIWA). Cylc is a workflow manager that automatically executes tasks according to the model's configuration. It also monitors all tasks, reporting any errors that may occur.
- The Rose framework developed by the UK Met Office (UKMO) which configures tasks for the Cylc engine. Rose is a toolkit that can be used to view, edit and run some of the ACCESS models.
A set of tasks configured by Rose to run with the Cylc7 engine is called a suite. Every suite has a unique identifier called suite-ID in the form u-LLNNN, where L is a letter and N is a number (e.g., u-ab123).
Prerequisites
-
NCI account
Before running an ACCESS model, you need to Set Up your NCI Account. -
MOSRS account
The Met Office Science Repository Service (MOSRS) is a server run by the UKMO to support collaborative development with other partners organisations. MOSRS contains the source code for some ACCESS model components and configurations, and a MOSRS account is a license requirement to run some ACCESS configurations.
To apply for a MOSRS account, please contact your local institutional sponsor.Warning
The waiting time to obtain a MOSRS account can be up to 3 weeks.
-
Join NCI projects
Join the following projects by requesting membership on their respective NCI project pages:
Connecting to Gadi
You can run Rose/Cylc either from a Gadi login node or via an ARE VDI session.
Connect via Gadi login node
Follow the steps to login to Gadi, making sure to enable X11 forwarding, for example by adding the -X option to the ssh command. This allows the Rose and Cylc GUIs to be launched on your local machine.
Once you are connected, skip directly to Set up a persistent session.
Launch ARE VDI Desktop
If you are not familiar with ARE, check out the Getting Started on ARE section.
Tip
The ARE VDI session does not run model tasks directly; it only runs Rose/Cylc. The model tasks are dispatched by Cylc to the compute nodes. This means the ARE VDI session requires minimal CPU and memory resources.
Go to the ARE VDI page and launch a session with the following entries:
-
Walltime (hours) →
2
Amount of hours the ARE VDI session will remain active for. This is only setup time, and does not reflect how long the actual configuration will take to run.Tip
Some model configurations might require a longer setup time. The Walltime included here should be sufficient for most model configurations, but if your ARE session terminates before the model setup is complete, you can start a new ARE VDI session.
-
Queue →
normalbw -
Compute Size →
tiny(1 CPU) -
Project → a project of which you are a member.
The project must have allocated Service Units (SU). By default, this will be set to your default project$PROJECT. -
Storage →
gdata/hr22+scratch/$PROJECT(minimum)
The storage folders listed above are the minimum required to run Rose/Cylc.
Launch the ARE session and, once it starts, click on Launch VDI Desktop.
Warning
This example is provided for reference only. Please use the resource specifications listed above when starting the ARE VDI session.
Once the new tab opens, you will see a Desktop with a few folders on the left. Click the terminal icon at the top of the window to open a terminal. You should now be connected to a Gadi computing node. Use this terminal for all subsequent steps in this guide.
Set up a persistent session
NCI provides a service called persistent sessions to enable long running processes, like Cylc, to stay active even when the user disconnects from Gadi.
It is recommended to have only one active persistent session at any given time, as multiple Cylc sessions can use the same persistent session.
Note that persistent sessions are terminated during the quarterly maintenance at NCI and will need to be restarted afterwards. The new persistent session can be given the same name as used previously.
Start a new persistent session
Start a new persistent session by running:
persistent-sessions start -p <project> <name>
where <project> is the project you want to start the session under, and <name> is the name you want to give your persistent session.
Warning
Persistent session names accept only a limited set of characters. We recommend using only alpha-numeric characters without spaces or underscores.
The label of a newly-created persistent session has the following format:
<name>.<$USER>.<project>.ps.gadi.nci.org.au.
Tip
If -p <project> is omitted, your default project $PROJECT will be used.
Tip
The project assigned to a persistent session does not have to be the same one used to run the ACCESS model configuration. In addition, the same persistent session can be used to run multiple simulations simultaneously.
The newly created persistent session is assigned a unique identifier, referred to here as <persistent-session-uuid>.
Assign the persistent session to Cylc (once only)
Once the session is running, it needs to be assigned to Cylc. This is done by inserting the persistent session label into ~/.persistent-sessions/cylc-session, which can be done with the following command (substituting <name> and <project> with the name and project used to create the persistent session).
cat > ~/.persistent-sessions/cylc-session <<< "<name>.${USER}.<project>.ps.gadi.nci.org.au"
You can check that this worked with:
cat ~/.persistent-sessions/cylc-session
For example, if user abc123 started a persistent session named ForCylc under the project tm70, then the command would be:
For more information on how to specify the target session, refer to Specify Target Session with Cylc7 Suites.
List active persistent sessions
To list your currently active sessions, use:
persistent-sessions list
Terminate a persistent session
To end a specific session, use:
persistent-sessions kill <persistent-session-uuid>
Tip
Logging out of a Gadi login node or an ARE VDI terminal instance will not affect your persistent session.
Warning
When you terminate a persistent session, any model running on that session will stop. Therefore, you should check whether you have any active model runs before terminating a persistent session.
Set up Rose/Cylc
Rose and Cylc executables
Make the rose and cylc executables available by loading the Cylc module:
module use /g/data/hr22/modulefiles
module load cylc7
MOSRS Authentication
The ACCESS models which use Cylc require a connection to the MOSRS mirror on Gadi. To connect to this mirror, you must first authenticate your MOSRS credentials with:
mosrs-auth
This will request the username and password you received when you created your MOSRS account.
After the first authentication, you will need to run mosrs-auth every 24 hours and for every new connection to Gadi (e.g., new terminal) to verify your password against the saved credentials.
Get the model configuration
Depending on the specific model, its configuration will be hosted either on GitHub or MOSRS. The Run a Model documentation for the respective model will specify where the configuration is stored.
Regardless of where the configuration comes from, it is recommended to store the local copy in the ~/roses/ directory (this happens automatically for configurations pulled from MOSRS). This directory will be referred to as the configuration directory.
This configuration directory contains multiple subdirectories and files, including:
app→ directory containing specific configuration files for various model tasks.meta→ directory containing the Rose GUI metadata.rose-suite.conf→ main model configuration file.rose-suite.info→ configuration information file.suite.rc→ Cylc control script file (Jinja2 language).
Model configurations stored on GitHub
For GitHub hosted configurations, get a local copy by cloning the GitHub repository with:
git -C ~/roses clone <repository> -b <branch>
where <repository> and <branch> are specific to the chosen model configuration and can be found in the respective Run a Model documentation.
If you want to make exploratory changes within the configuration, and have those changes tracked, please fork the configuration repository and commit your changes there.
If you think your new configuration provides significant value to the broader community, refer to the respective model configuration documentation for instructions on how to have it officially supported by ACCESS-NRI.
Model configurations stored on MOSRS
There are two ways of getting a local copy of a configuration hosted on MOSRS:
Both options use Rosie - a tool that uses SVN to manage, develop, and collaborate on Rose modelling workflows. It is automatically available within the Rose setup.
If you're not sure which option to use, we recommend using the "local-only copy". The "remote and local copy" is best used if you plan to commit the suite back to the remote.
Configurations copied from MOSRS are created by default in the user's Gadi home directory under ~/roses/<suite-ID> (this is the configuration directory).
Local-only copy
rosie checkout <suite-id>/<branch>
where <suite-id> and <branch> are specific to the chosen model configuration and can be found in the respective Run a Model documentation. This creates a local copy of the configuration, which is placed in the ~/roses/<suite-id> folder.
Tip
To copy from the default branch (trunk), omit the /<branch> portion of the command.
Configurations obtained in this way cannot be pushed back to the remote. Therefore, the use of this command is recommended for testing and examining configurations.
Local and remote copy (new remote configuration)
Before creating a new remote copy of the configuration, please read these guidelines on what should be stored in the Rosie repository.
rosie copy <suite-id>/<branch>
where <suite-id> and <branch> are specific to the chosen model configuration and can be found in the respective Run a Model documentation.
Tip
To copy from the default branch (trunk), omit the /<branch> portion of the command.
After running this command, a text editor will open in your terminal, where you can define metadata for the new configuration (the default text editor is Vim, and this quick guide is a good reference if you're unfamiliar with it). The metadata fields are expressed as key=value pairs, pre-filled with values copied from the original configuration. You can modify these values or add new metadata as needed. Note that owner, project and title are required keys.
owner=<MOSRS-username>
project=<project-name>
title=<suite-title>
When you exit the editor, you will have to confirm that you want to copy the suite:
This creates a new remote configuration with a new suite_id (based off the copied configuration) and clones a copy of it locally in the ~/roses/<new-suite-id> folder. Configurations created in this way are separate from the original copied configuration and can be modified and pushed back to the remote.
To push a configuration back to the remote, from within the configuration directory run:
fcm commit
For additional rosie options, run:
rosie help
Run the model configuration
Warning
Before running a configuration, make sure to follow the initial setup for it (e.g., setting the correct compute project and storage resources). For details, follow the instructions relative to your specific model in the Run a model page.
ACCESS model configurations run on Gadi through PBS jobs submissions. They often comprise several tasks, such as checking out code repositories, compiling and building different model components, running the model, etc. The workflow of these tasks is controlled by Cylc.
To run the configuration, execute the following commands:
rose suite-run -C ~/roses/<suite-id>
This launches the Rose/Cylc configuration and opens the Cylc GUI (if you are running on the login node and the GUI doesn't open, make sure you connected with X11 forwarding enabled). The Cylc GUI allows you to view and control the different tasks in the configuration as they run. The Cylc GUI can be safely closed without impacting the model run. If you closed the GUI and want to re-open it, run:
rose suite-gcontrol --name=<suite-id> &
Tip
The & is optional. It detaches the invoked process, allowing the terminal prompt to remain active while the GUI is open.
By default, the configuration, log files and outputs are copied to /scratch/<project>/${USER}/cylc-run/<suite-id>. A symbolic link to this directory is also created in your home directory under ~/cylc-run/<suite-ID>. See the respective Run a model documentation for details on what outputs are generated and where to find them.
Edit the model configuration
In general, the configurations of ACCESS models can be edited either by directly modifying the configuration files within the configuration directory, or by using the Rose GUI.
Rose GUI
To edit the model configuration, run the following:
rose edit -C ~/roses/<suite-id> &
This opens the Rose GUI, where the configuration settings can be modified. Once you are happy with the changes, save the modified configuration by clicking on the Save button. 
Tip
The & is optional. It detaches the invoked process, allowing the terminal prompt to remain active while the GUI is open.
Warning
Directly modifying configuration files with an editor is usually discouraged for non-expert users.
For a description of some common configuration settings, see the respective Run a model documentation.
You are now ready to a run a model with Rose/Cylc.