Run models using Rose/Cylc
Warning
ACCESS models configurations that run with Rose/Cylc currently use Cylc7, with intentions to upgrade to Cylc8. The upgrade is expected to change some aspects of the workflow described on this page. Updated information about the Cylc8 workflow will be provided once a model configuration using this version becomes available.
About
The Rose/Cylc workflow management tool consists of two components:
- The Cylc task engine, developed by the New Zealand National Institute of Water and Atmospheric Research (NIWA)
- The Rose framework developed by the UK Met Office (UKMO) which configures tasks for the Cylc engine.
A set of tasks configured by Rose to run with the Cylc7 engine is called a suite.
Prerequisites
-
NCI account
Before running an ACCESS model, you need to Set Up your NCI Account. -
MOSRS account
The Met Office Science Repository Service (MOSRS) is a server run by the UKMO to support collaborative development with other partners organisations. MOSRS contains the source code for some ACCESS model components and configurations, and a MOSRS account is a license requirement to run some ACCESS configurations.
To apply for a MOSRS account, please contact your local institutional sponsor. -
Join NCI projects
Join the following projects by requesting membership on their respective NCI project pages:
Connecting to Gadi
You can run Rose/Cylc either from a Gadi login node, or via an ARE VDI session. If you wish to use the Gadi login node, skip directly to Set up a persistent session.
X11 Forwarding
When connecting via SSH from a terminal, it is recommended to enable X11 forwarding, for example by adding the -X option to the ssh command. This allows the Rose and Cylc GUIs to be launched on your local machine.
Launch ARE VDI Desktop
Tip
The ARE VDI session does not run model tasks directly; it only runs Rose/Cylc. The model tasks are dispatched by Cylc to the compute nodes. This means the ARE VDI session requires minimal CPU and memory resources.
The following options are recommended for your ARE VDI desktop session:
-
Walltime (hours) →
2
Amount of hours the ARE VDI session will remain active for. This is only setup time, and does not reflect how long the actual configuration will take to run. -
Queue →
normalbw -
Compute Size →
tiny(1 CPU) -
Project → a project of which you are a member.
The project must have allocated Service Units (SU). By default, this will be set to your default project$PROJECT. -
Storage →
gdata/hr22+scratch/$PROJECT(minimum)
The storage folders listed above are the minimum required to run Rose/Cylc.
Once the ARE VDI session opens in your browser, click the terminal icon at the top of the window to open a terminal. Use this terminal for all subsequent steps in this guide.
Set up a persistent session
NCI provides a service called persistent sessions to enable long running processes, like Cylc, to stay active even when the user disconnects from Gadi.
It is recommended to have only one active persistent session at any given time, as multiple Cylc sessions can use the same persistent session.
Note that persistent sessions are terminated during the quarterly maintenance at NCI and will need to be restarted afterwards. The new persistent session can be given the same name as used previously.
Start a new persistent session
Start a new persistent session by running:
persistent-sessions start -p <project> <name>
where <project> is the project you want to start the session under, and <name> is the name you want to give your persistent session.
Warning
Persistent session names accept only a limited set of characters. We recommend using only alpha-numeric characters without spaces or underscores.
Tip
If -p <project> is omitted, your default project $PROJECT will be used.
Persistent sessions can run simulations using compute and storage resources from any project, independently of the project used for the persistent session itself. The newly created persistent session is assigned a unique identifier, referred to here as <persistent-session-uuid>.
Assign the persistent session to Cylc
Once the session is running, it needs to be assigned to Cylc. This is done by inserting the persistent session label into ~/.persistent-sessions/cylc-session, which can be done with the following command (substituting <name> and <project> with the name and project used to create the persistent session).
cat > ~/.persistent-sessions/cylc-session <<< "<name>.${USER}.<project>.ps.gadi.nci.org.au"
You can check that this worked with:
cat ~/.persistent-sessions/cylc-session
For example, if user abc123 started a persistent session named ForCylc under the project tm70, then the command would be:
List active persistent sessions
To list currently active sessions, use:
persistent-sessions list
Terminate a persistent session
To end a specific session, use:
persistent-sessions kill <persistent-session-uuid>
Rose/Cylc Setup
Rose and Cylc executables
Make the rose and cylc executables available by loading the Cylc module:
module use /g/data/hr22/modulefiles
module load cylc7
MOSRS Authentication
The ACCESS models which use Cylc require a connection to the MOSRS mirror on Gadi. To connect to this mirror, you must first authenticate your MOSRS credentials with:
mosrs-auth
This will request the username and password you received when you created your MOSRS account.
After the first authentication, you will need to run mosrs-auth every 24 hours and for every new connection to Gadi (e.g., new terminal) to verify your password against the saved credentials.
Get the model configuration
Depending on the specific model, its configuration will be hosted either on GitHub or MOSRS. The Run a Model documentation for the respective model will specify where the configuration is stored.
Regardless of where the configuration comes from, it is recommended to store the local copy in the ~/roses/ directory (this happens automatically for configurations pulled from MOSRS).
Model configurations stored on GitHub
For GitHub hosted configurations, get a local copy by cloning the GitHub repository with:
git -C ~/roses clone <repository> -b <branch>
where <repository> and <branch> are specific to the chosen model configuration and can be found in the respective Run a Model documentation.
If you want to make exploratory changes within the configuration, and have those changes tracked, please fork the configuration repository and commit your changes there.
If you think your new configuration provides significant value to the broader community, refer to the respective model configuration documentation for instructions on how to have it officially supported by ACCESS-NRI.
Model configurations stored on MOSRS
There are two ways of getting a local copy of a configuration hosted on MOSRS:
Local-only copy
rosie checkout <suite-id>/<branch>
where <suite-id> and <branch> are specific to the chosen model configuration and can be found in the respective Run a Model documentation. This creates a local copy of the configuration, which is placed in the ~/roses/<suite-id> folder.
Tip
To copy from the default branch (trunk), omit the /<branch> portion of the command.
Configurations obtained in this way cannot be pushed back to the remote. Therefore, the use of this command is recommended for testing and examining configurations.
Local and remote copy (new remote configuration)
Before creating a new remote copy of the configuration, please read these guidelines on what should be stored in the Rosie repository.
rosie copy <suite-id>/<branch>
where <suite-id> and <branch> are specific to the chosen model configuration and can be found in the respective Run a Model documentation.
Tip
To copy from the default branch (trunk), omit the /<branch> portion of the command.
After running this command, a text editor will open in your terminal, where you can define metadata for the new configuration (the default text editor is Vim, and this quick guide is a good reference if you're unfamiliar with it). The metadata fields are expressed as key=value pairs, pre-filled with values copied from the original configuration. You can modify these values or add new metadata as needed. Note that owner, project and title are required keys.
owner=<MOSRS-username>
project=<project-name>
title=<suite-title>
When you exit the editor, you will have to confirm that you want to copy the suite:
This creates a new remote configuration with a new suite_id (based off the copied configuration) and clones a copy of it locally in the ~/roses/<new-suite-id> folder. Configurations created in this way are separate from the original copied configuration and can be modified and pushed back to the remote.
To push a configuration back to the remote, from within the configuration directory run:
fcm commit
Run the model configuration
To run the configuration, execute the following command from within the configuration directory:
rose suite-run
This launches the Rose/Cylc configuration and opens the Cylc GUI (if you are running on the login node and the GUI doesn't appear, make sure you connected with X11 forwarding enabled). If you closed the GUI and want to re-open it, navigate to the configuration directory and run:
rose suite-gcontrol &
Tip
The & is optional. It detaches the invoked process, allowing the terminal prompt to remain active while the GUI is open.
By default, the configuration, log files and outputs are copied to /scratch/<project>/${USER}/cylc-run/<suite-id>. See the respective Run a model documentation for details on what outputs are generated and where to find them.
Edit the model configuration
To edit the model configuration, run the following command from the configuration directory:
rose edit &
This opens the Rose GUI that contains information about the configuration settings. For a description of the simple configuration settings, see the respective Run a model documentation.
Once settings have been modified in the Rose GUI, save them by clicking on the Save button 
.
You are now ready to a run a model with Rose/Cylc.