1. Introduction
2. Foundations
3. Git and GitHub
4. Accessing Remote Systems via the SSH Protocol
5. Getting Started With the SRW App
6. Running the SRW App on AWS
7. Setting Up the SRW App on AWS
1. Introduction
Introduction
What is the UFS?
The Unified Forecast System (UFS) is a community-based, coupled, comprehensive Earth modeling system. NOAA’s operational model suite for numerical weather prediction (NWP) is quickly transitioning to the UFS from a number of different modeling systems. The UFS enables research, development, and contribution opportunities within the broader Weather Enterprise (including government, industry, and academia).What Will This Guide Walk You Through?
This guide will:- Cover foundational knowledge related to the Linux command line, Git/GitHub, and cloud computing and
- Walk users through setting up and running the UFS Short-Range Weather (SRW) Application (App), with specific instructions depending on your computing environment (i.e., Mac, Linux, Windows).
2. Foundations
Foundations
The UFS can run on any reasonably up-to-date UNIX-style operating system; in other words, the UFS can run natively on a Linux or Mac system. Windows users have several options available, including running on a virtual machine or in the cloud. Whatever option users choose, they will need to know some command-line basics to perform different tasks.Command Line Basics
Navigating the File System
These commands help users understand and move within the file system:-
pwd: Print working directory. -
cd: Change directory. -
ls: List directory contents.
Understand Paths
- Absolute Paths: Begin from the root directory (/), e.g.,
/home/user/documents - Relative Paths: Begin relative to the current directory.
- The period (.): Current directory.
- Two periods (..): The directory above the current directory.
/home/user/documents, a user who wants to navigate back to /home/user can use the command cd ..
Basic Commands and File Permissions
- Viewing Files:
catallows users to display the contents of a file. For viewing large files, less and more allow users to scroll through the file content. - Editing Files:
vim, nano,
andemacs
are all different command line editors. Each of these file editors is suited for different user preferences, although vim is the default for many users, since it comes preinstalled on most Linux systems.
Shell Environment and Variables
When users open a Terminal window, a shell is launched. The shell is a program that processes user commands and returns output. Bash is one of the most common shells and the assumed default in UFS applications, but other shells exist (e.g., csh/tcsh). The shell environment contains shell settings, including user-defined settings, often stored in a .bashrc or .bash_profile file. Users can also set environment variables to store small pieces of information, such as file paths or other user-defined values, on a system. These are typically available only in the shell where they are set, but environment variables set in a .bashrc (or similar) login file will be available in any shell the user opens on a given machine. Modules or modulefiles can be used to load related settings required for specific applications. Modulefiles often set environment variables when they are loaded and then unset them when they are unloaded.- Environment Variables:
$PATH, $LIBRARY_PATH, $LD_LIBRARY_PATH - Configuring Shell Environment:
.bashrc, .bash_profile - Modules:
module use/module load
Getting Help
Theman command can be run to find out more about what a particular command does. For example, man ls will print information on what the ls command does and what options can be used with it. Many scripts can be run with the -h option to find out how to use the script (e.g., ./generate_FV3LAM_wflow.py -h).
- Manual Pages:
man - Help Flags:
--help, -h
Introduction to Shell Scripting
Users can edit a script using their command line editor. For example, withvim,
- Run :
vim myscript.shto :open myscript.sh. This command will create the file if it doesn’t exist yet. - To edit a file, press the
ikey for “insert” mode after opening the file. - Press
escto exit insert mode. - Type
:wqto save (or “write”) the file and quit the editor. - Run the script:
./myscript.sh
3. Git and GitHub
Git and GitHub
Git and GitHub are version control tools that allow users to keep track of changes to their code. To keep track of changes and updates to files, programmers use Git. Think of Git as a computer code equivalent of the Track Changes feature in Microsoft Word. Git allows programmers to see changes in files, see who made the changes, and view/return to old versions of a file. To check whether Git is installed on your system, open a terminal window and run:
git --version
If it does not print out a version number (e.g., git version 2.39.3 (Apple Git-146)), you may need to install Git.
GitHub is a web-based service for Git. Although there are other GitHub-like solutions (e.g., GitLab), all UFS code is publicly available for free on GitHub, and development is coordinated through GitHub. Users can save (or “commit”) code changes on their system using Git and then send (or “push”) the changes to GitHub. If their computer crashes and they lose all of their data, their project is safe on GitHub, and they can retrieve the project code and put it on a new (or repaired) system later. In addition to this backup functionality, GitHub allows collaboration and version tracking between different project contributors. The contribution process is described in the UFS wiki and in various UFS documentation.
The EPIC program has created two tutorials on Git/GitHub Basics. Users are encouraged to view the accompanying slides for basic Git terms, concepts, and commands.
- GitHub Tutorial: Part 1. Contributing to UFS/EPIC Repositories
- GitHub Tutorial: Part 2. Contributing to UFS/EPIC Repositories
Creating a GitHub Account
UFS developers will need a free GitHub account to contribute to UFS repositories. Users can copy (or “clone”) UFS code without an account, but anyone interested in modifying code for their own experiments will likely benefit from a GitHub account. To get started, navigate to www.github.com and click on “Sign Up” in the top right hand corner. Follow the prompts to create an account. Community members can refer to this guide if they run into problems. After creating an account, it is recommended that users and developers complete the following steps:- Set up two-factor authentication under Settings -> Password and authentication in their GitHub account.
- Configure GitHub to connect via SSH (secure shell protocol): GitHub has a comprehensive guide to this process. Users who already have an SSH key can add an SSH key to their GitHub account under Settings -> SSH and GPG keys.
4. Accessing Remote Systems via the SSH Protocol
Accessing Remote Systems via the SSH Protocol
The Secure Shell (SSH) protocol is a way to connect to remote systems securely. The SSH protocol uses a public key and a private key for authentication. To generate a public/private SSH key pair:-
- Open a terminal window and navigate to ~/.ssh
- Run
ssh-keygen -t rsato generate a key pair. - Hit enter three times to accept defaults, or if customization is desired:
- Enter the full path to the file in which to save the key (for example:
~/.ssh/id_rsa) - Enter passphrase (Leave empty/hit ”Enter” for no passphrase)
- Enter same passphrase again
- Enter the full path to the file in which to save the key (for example:
- To see the public key, run:
cat id_rsa.pub
- Start a new SSH agent:
eval "$(ssh-agent -s)" - Add a new private key to the SSH agent:
ssh-add ~/.ssh/id_rsa
SSH Authentication for GitHub
For help with using SSH keys on GitHub, visit GitHub’s SSH Documentation.Using SSH to Connect to a Remote System
This process differs somewhat from system to system, and users who have access to an HPC system should contact their system administrator for instructions on how to connect. Once users have an SSH key pair on their system, they can use the ssh command to log in to their remote system (e.g., HPC, AWS instance). This command follows the pattern:ssh [options] host [command]More concretely, this will look like:ssh username@remote-server.comIn the SRW App on AWS tutorial below, users will see a command that includes the SSH public key file:ssh -i “EPIC_RSA.pem” ec2-user@ec2-34-205-156-33.compute-1.amazonaws.comIn reality, users will need to enter the full path toEPIC_RSA.pemin the command if the key is not located in their terminal window’s current working directory. Running this command will create an SSH tunnel between the user’s local system and the AWS cluster they set up. The user will be able to work on the AWS cluster by running commands in their computer’s terminal window. In some cases, the user may be asked if they want to connect:The authenticity of host 'ec2-user@ec2-34-205-156-33.compute-1.amazonaws.com (34.205.156.33)' can't be established. ED25519 key fingerprint is SHA256:a0ABbC4cdeDEfFghi+j3kGHlO5mnIJKLMop7NOqPrQR. Are you sure you want to continue connecting (yes/no/[fingerprint])?
Enteryesto continue connecting. The user is responsible for verifying that they are connecting to the correct system.
5. Getting Started With the SRW App
Getting Started With the SRW App
What is the SRW App?
The UFS Short-Range Weather (SRW) Application (App) is a UFS application that targets predictions of atmospheric behavior on a limited area grid and on time scales from minutes to several days. Users can explore resources for the SRW App on the EPIC website.
Where Can You Run the SRW App?
- NOAA RDHPCS: If you have access to a NOAA Research & Development High-Performance Computing System (RDHPCS), the SRW App User’s Guide includes all of the information you need to get started. The Quickstart chapter provides a brief summary of how to build and run the SRW App; these instructions are designed to work well for users on NOAA RDHPCS and are a good starting place. The Introduction chapter lays out the contents of the User’s Guide and may help users determine what other sections will be useful to them.
- University or Private Industry HPCs: Users can choose to run the SRW App in a container, or they can set up the prerequisite software and create a machine file for their environments. This option is not yet fully supported, but the EPIC program will attempt to provide assistance to users who have questions.
- No Access to an HPC? The walkthrough below will explain how to run the SRW App on the cloud. Although there are many cloud service providers available, it is not practical to provide instructions for all of them. Therefore, this walkthrough will use AWS.
6. Running the SRW App on AWS
Running the SRW App on AWS
Introduction to Cloud Computing and AWS
Cloud computing delivers computing services—servers, storage, databases, computing power, networking, software, analytics, and more—over the internet (“the cloud”). UFS applications and models often require more computational power than the average personal computer has, and not everyone has access to HPC systems. Cloud computing can fill this gap; anyone can sign up for an account with a cloud service provider such as Amazon Web Services (AWS) and set up a system that can run the UFS.
Core AWS Services
AWS is a comprehensive and widely adopted cloud platform. AWS offers over 200 fully-featured services globally, including:
- EC2 (Elastic Compute Cloud): A virtual server in the AWS Cloud that provides a scalable computing environment. There are several instance types, but this tutorial series uses compute-optimized instances.
- S3 (Simple Storage Service): Data storage with a simple web service interface.
- VPC (Virtual Private Cloud): Enables users to launch AWS resources in a virtual network.
Creating an AWS Account
To start using AWS, you need to create an AWS account:
- Visit the AWS Signup page.
- Follow the on-screen instructions to create a new account.
7. Setting Up the SRW App on AWS
Setting Up the SRW App on AWS
Creating a Base Image on AWS
Before running the SRW App on AWS, users will need to set up an EC2 instance. An Amazon EC2 instance provides a virtual server and Amazon Machine Images (AMIs), which are “[p]reconfigured templates for your instances that package the components you need for your server (including the operating system and additional software).” In other words, a properly configured EC2 instance provides the hardware and software required to mimic a physical computer. Users can watch the video tutorial Creating a Base Image on Amazon Web Services (AWS) for step-by-step instructions on how to create an EC2 instance. The video shows how to:- Launch an AWS EC2 instance.
- Create an SSH key pair for secure authentication.
- ssh into the EC2 instance you created.
- Install prerequisites: Git, yum-utils, yum-config-manager, Packer.
- Download (or “clone”) the packer-srwcluster repository, which contains all the scripts needed to install SRW App prerequisite software, such as spack-stack.
Launching a PCluster Image to Run the SRW App on AWS
A ParallelCluster (or PCluster) is an AWS tool for managing high-performance computing (HPC) resources in the AWS cloud. After following the steps to set up a base image and download the packer-srwcluster code, users can watch the video tutorial Launching a PCluster Image to run SRW on Amazon Web Services (AWS) for step-by-step instructions on how to launch the computing resources needed to run an SRW App forecast. This video shows how to:- Modify and run the scripts used to install PCluster and a few additional SRW App prerequisites (e.g., Lmod).
- Create an S3 bucket for data storage.
- Launch a HeadNode, or “login node,” which is the part of an HPC system where users access the system to perform various tasks, such as managing files, editing scripts, compiling code, and submitting jobs to the compute nodes.
- Log in to the HeadNode.
Running the SRW App
Currently, the most recent instructions for running the SRW App in the cloud come from the UIFCW Short-Range Weather App Training on AWS for New Users. These instructions are from July 2024. Users can also refer to the SRW App v2.2.0 Container-Based Quick Start Guide if they prefer to build the SRW App v2.2.0 release (from October 2023). This chapter of the User’s Guide outlines steps for building and running the SRW App v2.2.0 in a container to get users started with their first forecast experiment. Regardless, it is recommended that users build the SRW App from a container. Normally, the details of building and running the UFS applications and models vary from system to system due to the many possible combinations of operating systems, compilers, MPIs, and package versions available. However, users can take advantage of containers to run UFS systems more easily. Docker describes a container as “a standard unit of software that packages up code and all its dependencies so the application runs quickly and reliably from one computing environment to another.” If users choose to build the SRW App without a container, they will need to create modulefiles (underufs-srweather-app/modulefiles) and a machine file (under ufs-srweather-app/ush/machine) that include the details of their EC2 configuration (e.g., spack-stack installation location, Rocoto installation location). However, this option is currently unsupported and not ideal for beginners.