AIMIP Phase 1: systematic evaluations of AI weather and climate models

We present the AI weather and and climate model intercomparison project (AIMIP), phase 1. Drawing from the rich tradition of model intercomparisons in climate model development, we specify a common experiment, output data format, and training constraints (namely, training against historical reanalysis data) for AIMIP Phase 1 models. We aim to identify differences between modeling frameworks and AI architectural choices that influence model behavior, and build trust in AI weather and climate models through open data and evaluation. AIMIP Phase 1 models must simulate the atmosphere given specified historical sea surface temperatures over 1979-2024; we then evaluate the models’ performance using five major evaluation criteria: biases, trends, response to El Niño-related sea surface temperature anomalies, temporal variability, and out of sample generalization tests. We find that AI models are able to simulate the historical climate and response to forcing as well as a conventional physically-based model, but some AI models underestimate historical warming trends, and their predictions diverge in the out of sample generalization tests. We describe the AIMIP Phase 1 dataset that is publicly available for additional
evaluations.