Training Resources

GitHub Tutorial: Part 1. Contributing to UFS/EPIC Repositories

This video requires some basic linux command line knowledge ahead of time. It will talk you through the process of learning the basics of Git, an GitHub integrations.

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

This tutorial is the first of two tutorials that cover the basics of Git and GitHub, including common terms, concepts, and commands. It demonstrates how to:

• Create repositories, forks, and branches (local and remote)
• Use basic Git commands
• Commit (save) changes locally and push them to a remote repository
• Explore remote repositories

This Git/GitHub tutorial is intended to facilitate collaborative code development and to promote contributions from UFS community developers to the UFS and NOAA-EPIC GitHub repositories.

GitHub Tutorial: Part 2. Contributing to UFS/EPIC Repositories

This video requires some basic linux command line knowledge ahead of time. It will talk you through the process of learning the basics of Git, an GitHub integrations.

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

It is highly encouraged to watch Part 1 of the tutorial before proceeding with this video.

This tutorial is Part 2 of the series on contributing to UFS/EPIC repositories. It presents instructions on navigating local and remote repositories using Git and GitHub.  The instructions build on the material presented in the Part 1 tutorial, which covered Git and GitHub basic terms and operations. The following information is covered in Part 2: 

• UFS/EPIC public repositories
• Forks and clones of the repositories
• Branches and tags
• Steps to contribute to open-source repositories
• Three phases of local Git space
• Fetching and merging remote branches
• Resolve merge conflicts 
• Making pull requests (PRs)
• Testing another developer’s PR and providing feedback
• GitHub Issues and Discussions pages for support

UFS Weather Model Test Case Walkthrough

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

The UFS Weather Model (WM) includes a developmental testing framework designed to facilitate the integration of hierarchical testing and development practices into the UFS ecosystem. 

This video walks through how to run two WM test cases on the Mississippi State University (MSU) Hercules supercomputer. The steps are repeatable on other supported Level 1 systems (e.g., NCAR Derecho and NOAA’s Hera machine). Users on non-Level 1 systems may still benefit from watching this video but should refer to the section of the User’s Guide for running UFS WM idealized cases in a container. This video covers:

• Useful resources for the UFS WM
• Workflow steps/tasks

Users will also see many basic Linux and Git-related commands in use. Visualization scripts are available in utils directory of tests-dev, providing users with options to plot various fields of interest for each case. Also, available in this directory is a README file with brief instructions on how to run the plotting scripts.

Land DA Workflow Demo

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

In the Land Data Assimilation (DA) System, the Noah-MP land surface model (LSM) from the UFS Weather Model (WM) and the Joint Effort for Data assimilation Integration (JEDI) system are used to assimilate snow depth data via the Local Ensemble Transform Kalman Filter-Optimal Interpolation (LETKF-OI) algorithm. 

This video walks through how to run the Land DA System on the Mississippi State University (MSU) Orion supercomputer. The steps are repeatable on other supported Level 1 systems (i.e., MSU Hercules, and NOAA’s Hera machine). Users on non-Level 1 systems may still benefit from watching this video but should refer to the section of the User’s Guide for running Land DA in a container. This video covers:

• Useful resources for Land DA
• Workflow steps/tasks
• Land DA directory structure

Users will also see many basic Linux and Git-related commands in use. 

Building & Running the Weather Model Using Containers on a Generic System

Mark Potts will demonstrate how the latest container, based on spack-stack v1.8.0, can be used to build and run the Global Workflow on a generic Linux system. The focus will be on forecast-only tasks, with potential for cycling and ensemble extensions.

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description:

The following steps are designed to show users how to build and run the containerized Global Workflow. The user has the option to build a global-workflow container from existing Docker images, copy the container from NOAA Tier-1 Research and Development High-Performance Computing Systems (RDHPCS), or download the container image directly from the AWS S3 storage bucket. Because the container is already built using spack-stack v1.8.0, it eliminates the need to build a separate spack-stack environment on the platform and simplifies the process of porting the Global Workflow to Tier-1 machines. The container also allows non-NOAA researchers to work directly with UFS applications.

Containerized SRW App v3.0.0 Walkthrough

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description:

The video will help users build and run the “out-of-the-box” case for the Unified Forecast System (UFS) Short-Range Weather (SRW) Application using a Singularity/Apptainer container. The container approach provides a uniform environment in which to build and run the SRW App. Normally, the details of building and running the SRW App vary from system to system due to the many possible combinations of operating systems, compilers, MPIs, and package versions available. Installation via container reduces this variability and allows for a smoother SRW App build experience.

This video covers:

• Useful resources for the containerized version of SRW App
• Workflow steps/tasks

Users will also see many basic Linux commands in use. 

SRW App v3.0.0 Plots and Postprocessing Walkthrough

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

The video will help users build and run the “out-of-the-box” case for the Unified Forecast System (UFS) Short-Range Weather (SRW) Application v3.0.0 on a Tier-1 platform. The first session of this tutorial focuses on building and running the SRW App cloned from the UFS GitHub repository.
In the second and third sessions, we’ll walk through the setup process and then dive into a detailed exploration of the output data and image files generated by the forecast experiment.

The three Key sections of this tutorial are as follows, and section 1 applies to the non-containerized version only, and sections 2 & 3 apply to both containerized and non-containerized versions

• Cloning SRW App from EPIC GitHub Repository & Building it on Tier-1 Platforms
• Model Configuration Setup
• Outputs and Plot Walkthrough

This video covers:

• Useful resources for both containerized and non-containerized versions of SRW App
• Workflow steps/tasks

Users will also see many basic Linux commands in use. 

Creating a Base Image on Amazon Web Services (AWS)

This video requires that you have an AWS account, and have limited knowledge of EC2 and S3. This video will create an AWS AMI that can then be generated into an HPC resource.

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

This instructional video will walk users through how to set up an Amazon Web Services (AWS) Amazon Machine Image (AMI) that will allow them to run the Short-Range Weather (SRW) Application. It installs and builds everything needed to run the SRW Application. Users can also leverage the multi-cloud Packer framework to build out other application images by running a handful of commands.

Launching a PCluster Image to run SRW on Amazon Web Services (AWS)

This video requires that you have an AWS account, and have limited knowledge of EC2 and S3. This video will create the AWS HPC resources needed to build the SRW application.

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

This instructional video will walk users through how launch a SRW-configured image on an Amazon Web Services (AWS) Amazon Machine Image (AMI) that will allow them to run the Short-Range Weather (SRW) Application. It installs and builds everything needed to run that was loaded in the prior video. The last tutorial in this series will walk users through running the SRW Application end-to-end.

Building and Running Containerized Land Data Assimilation (DA) Workflow on Amazon Web Services (AWS)

This instructional video will walk users through how to build and run the containerized Land DA System using the AWS HPC resources built in Tutorials 1 and 2. 

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description:

This is the third tutorial in the Cloud series for running UFS applications in AWS. This video requires that you have an AWS account and limited knowledge of EC2 and S3. This video will allow you to build and run the containerized Land DA workflow from the AWS HPC resources built in Tutorials 1 and 2.

An Introduction to Unified Workflow Tools and the File Copy/Link Tool

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

The uwtools Python package contains generic tools and Unified Forecast System (UFS) software component drivers that facilitate configuring and running numerical weather prediction applications. This introduction covers the basics of installing uwtools and configuring and running a Unified Workflow (UW) tool on the command line and using the API.

This video corresponds to uwtools v2.3.x.

Visit the GitHub Repository: https://github.com/ufs-community/uwtools

Visit our Documentation: https://uwtools.readthedocs.io/en/main/

HPC-Stack Setup on a Mac

The key to this video is to build HPC-Stack on a Macbook Pro. HPC-Stack will soon be replaced by Spack-Stack, but this is a key different requirement to run locally on a Mac for the time being.

**Please be aware that, due to potential conflicts with Intel’s end-user licensing agreement, all content and repositories related to EPIC containers have been made private. This impacts our spack-stack, SRW, Land DA, and HSD containers. We apologize for any inconvenience and are actively working toward a solution.**

Description

This video will guide users through the process of installing and building HPC-Stack on MacOS systems with M1/arm64 or x86_64 architecture. 

HPC-Stack provides a unified, shell script-based build system to build the software stack required for numerical weather prediction (NWP) tools such as the Unified Forecast System (UFS) and the Joint Effort for Data assimilation Integration (JEDI) framework.

This tutorial covers the following steps:

• Preparing your MacOS system by installing prerequisites (e.g., Homebrew, Xcode Command Line Tools (CLT), compilers, CMake, Make, OpenSSL, Lmod, Wget, Python, and Git)
• Cloning the High-Performance Computing Stack (HPC-Stack) repository
• Configuring the system (e.g., specifying compilers, Python libraries, Message Passing Interface (MPI) libraries, and appropriate flags)
• Setting up the Lmod environment and customizing the HPC-Stack installation by specifying which MPI libraries to build.
• Building the HPC-Stack, setting up modules and environment, and finally, compiling the software stack.

Join us as we walk you through each step to successfully install and build HPC-Stack on your MacOS system.