As we extend the forecast-verification figures, we have added comparison plots between conventional, physics-based GFS and HRRR forecast systems and AI/data-driven nested-EAGLE and AIGFS. Details for each forecast system and its publicly available data can be found here: GFS, HRRR, nested-EAGLE, and AIGFS. The following nested-EAGLE model definitions will be used: Nested-EAGLE Global is the global domain of the global-nested EAGLE product and Nested-EAGLE CONUS is the CONUS domain of the global-nested EAGLE product. The GFS, HRRR, and AIGFS are on their native grids.
RMSE difference plots are computed for GFS, HRRR, nested-EAGLE, and AIGFS forecast systems against the AI-ready archive of conventional observations using 4-times daily, 10-day retrospective forecasts for 2025 (i.e., 1,460 forecasts total per system). To compare gridded forecast products to observations, the Earth System Modeling Framework is used to bilinearly interpolate the gridded values to the closest observation point to compute statistics. RMSE differences are displayed for the global, Northern hemisphere, Southern hemisphere, and CONUS domains.
The RMSE improvement is computed, for example, as
100 × (GFS RMSE − Nested-EAGLE RMSE) / GFS RMSE.
Verification includes 3-D atmospheric variables (geopotential height, zonal-, meridional-, and total-wind speed, temperature, and specific humidity) at 250, 500, and 850 hPa, along with selected surface variables (2-m temperature, 2-m specific humidity, and 10-m zonal-, meridional-, and total-wind speeds).
Retrospective Forecast Verification
Start date: 2024-02-01 00Z
End date: 2025-01-31 18Z
Initializations every 30 hours.
These plots are associated with model runs staggered across different forecast cycles (00Z, 06Z, 12Z, and 18Z) approximately every fourth day to capture varying points within the diurnal cycle. As a result, the displayed plots are static and represent model runs performed slightly less than once per day over the course of a full year. The shading in each plot represents the bootstrapped 95% confidence interval around the estimated median at each lead time.
Retrospective Forecast Verification Scorecards
Global, Northern Hemisphere, Southern Hemisphere, and CONUS (left to right) RMSE differences (percentage improvement) of nested-EAGLE global domain relative to GFS for forecast lead times from day 1 through day 10 (i.e., d1 through d10). Positive (blue) values indicate lower RMSE (improved forecast skill) for nested-EAGLE relative to GFS, while negative (red) values indicate higher RMSE. Numbers within each cell denote the RMSE change.
Global, Northern Hemisphere, Southern Hemisphere, and CONUS (left to right) RMSE differences (percentage improvement) of nested-EAGLE global domain relative to AIGFS for forecast lead times from day 1 through day 10 (i.e., d1 through d10). Positive (blue) values indicate lower RMSE (improved forecast skill) for nested-EAGLE relative to AIGFS, while negative (red) values indicate higher RMSE (decreased forecast skill). Numbers within each cell denote the RMSE change. Note that nested-EAGLE is trained on 8 years of GFS and HRRR data, while AIGFS is trained on 45 years of ERA5 data with GDAS refinement. The increased performance of AIGFS demonstrates the need for extending the training period for nested-EAGLE in the future. This figure is for illustrative purposes only.
CONUS RMSE differences (percentage improvement) of nested-EAGLE CONUS domain relative to HRRR for forecast lead times from hour 0 through hour 48 (i.e., 0h through 48h). Positive (blue) values indicate lower RMSE (improved forecast skill) for the nested-EAGLE relative to HRRR, while negative (red) values indicate higher RMSE (decreased forecast skill). Numbers within each cell denote the RMSE change.
Near real-time Verification
To complement the retrospective scorecards above, we also compute verification against AI-ready archive of conventional observations for both the nested-EAGLE global and CONUS domains in near-real time (i.e., verification performed on a 12-day delay to allow for RMSE and ME calculation). Here we include 3-D atmospheric variables (geopotential height, zonal- and meridional-wind speed, temperature, and specific humidity) at 250, 500, and 850 hPa along with selected surface variables (2 m temperature, surface pressure, and 10 m zonal- and meridional-wind speeds). Users can select the desired variable, pressure level, domain, and metric for a customized evaluation.
