ecmwf/aifs-ens-1.0

AIFS ENS - v1.0

Here, we introduce the Artificial Intelligence Forecasting System (AIFS) ENS, an ensemble-based probablistic data driven forecast model developed by the European Centre for Medium-Range Weather Forecasts (ECMWF).

AIFS produces highly skilled forecasts for upper-air variables, surface weather parameters and tropical cyclone tracks. AIFS ENS is run four times daily alongside ECMWF’s physics-based NWP model and forecasts are available to the public under ECMWF’s open data policy. Note that due to the non-determinism of GPUs, users will be unable to exactly reproduce an official AIFS ENS forecast when running AIFS ENS themselves.

For more details please refer to our documentation.

Data Details

Data parameters

Note Regarding precipitation units, it's worth noting that AIFS model was trained on $m^3\mathrm{/}{m}^{2}m^\{3\}/m^\{2\}$ and will therefore produce precip in that units. If one wants to retrieve precipitation from Open data, the units will be $mmmm$.

Model Details

Description

For full details of the AIFS ENS, please see Lang et al. 2024.

AIFS is based on a graph neural network (GNN) encoder and decoder, and a sliding window transformer processor, and is trained on ECMWF’s ERA5 re-analysis and ECMWF’s operational numerical weather prediction (NWP) analyses.

It has a flexible and modular design and supports several levels of parallelism to enable training on high resolution input data. AIFS forecast skill is assessed by comparing its forecasts to NWP analyses and direct observational data.

AIFS ENS was trained to minimise the Continuous Ranked Probability Score (CRPS) calculated over a small ensemble.

• Developed by: ECMWF
• Model type: Encoder-processor-decoder model
• License: These model weights are published under a Creative Commons Attribution 4.0 International (CC BY 4.0). To view a copy of this licence, visit https://creativecommons.org/licenses/by/4.0/. The notebooks and other script files are published under an Apache 2.0 licence, to view a copy of this license, visit https://www.apache.org/licenses/LICENSE-2.0.txt.

Resolution

	Component	Horizontal Resolution [kms]	Vertical Resolution [levels]
Atmosphere	AIFS ENS v1.0	~ 31	13

Sources

• Repository: Anemoi is an open-source framework for creating machine learning (ML) weather forecasting systems, co-developed by ECMWF and a range of national meteorological services across Europe.
• Paper: https://arxiv.org/abs/2412.15832

Get started

🚨 Note aifs-ens-1.0 requires 38GB of GPU memory by default, it is possible to reduce this by setting ANEMOI_INFERENCE_NUM_CHUNKS to a multiple of 2. For example setting it to 16 reduces the memory requirements to 24GB at the cost of throughput. Therefore issues may arise with lower end GPUs.

To generate a new forecast using AIFS ENS, you can use anemoi-inference. In the example notebook, a step-by-step workflow is specified to run AIFS ENS using the HuggingFace model:

1. Install Required Packages and Imports
   • torch==2.5.0 anemoi-inference[huggingface]==0.6.0 anemoi-models==0.6.0 anemoi-graphs==0.6.0 anemoi-datasets==0.5.23
   • earthkit-regrid==0.4.0 'ecmwf-opendata>=0.3.19'
   • flash_attn
2. Retrieve Initial Conditions from ECMWF Open Data
   • Select a date
   • Get the data from the ECMWF Open Data API
   • Get input fields
   • Add the single levels fields and pressure levels fields
   • Convert geopotential height into geopotential
   • Create the initial state
3. Load the Model and Run the Forecast
   • Download the Model's Checkpoint from Hugging Face
   • Create a runner
   • Run the forecast using anemoi-inference
4. Inspect the generated forecast
   • Plot a field

🚨 Note we train AIFS using flash_attention. The use of the 'Flash Attention' package also imposes certain requirements in terms of software and hardware, which can be found under Installation and Features on the GitHub repo.

🚨 Note the aifs_ens_v1.0.ckpt checkpoint just contains the model’s weights. It does not contain any information about the optimizer states, lr-scheduler states, etc.

Citation

If you use this model in your work, please cite it as follows:

BibTeX:

@misc{lang2024aifscrps,
      title={AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score}, 
      author={Simon Lang and Mihai Alexe and Mariana C. A. Clare and Christopher Roberts and Rilwan Adewoyin and Zied Ben Bouallègue and Matthew Chantry and Jesper Dramsch and Peter D. Dueben and Sara Hahner and Pedro Maciel and Ana Prieto-Nemesio and Cathal O'Brien and Florian Pinault and Jan Polster and Baudouin Raoult and Steffen Tietsche and Martin Leutbecher},
      year={2024},
      eprint={2412.15832},
      archivePrefix={arXiv},
      primaryClass={physics.ao-ph},
      url={https://arxiv.org/abs/2412.15832}, 
}

APA:

Lang, S., Alexe, M., Clare, M. C., Roberts, C., Adewoyin, R., Bouallègue, Z. B., ... & Leutbecher, M. (2024). AIFS-CRPS: Ensemble forecasting using a model trained with a loss function based on the Continuous Ranked Probability Score. arXiv preprint arXiv:2412.15832.

More Information

Find the paper here