# VA-MoE: Variables-Adaptive Mixture of Experts for Incremental Weather Forecasting

arXiv: https://arxiv.org/abs/2412.02503 · HTML: https://mlsanity.com/paper/2412.02503/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts.
- Builds on 26 of its 50 references that are in the map.
- First posted on arXiv 3 Dec 2024.

## Placed in

- Mixture-of-experts / [MoE for time series and graphs](https://mlsanity.com/moe/moe-for-time-series-and-graphs/)

## Builds on (most cited first, 26 of 26)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [Expert Gate: Lifelong Learning with a Network of Experts](https://mlsanity.com/paper/1611.06194/) (2016)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [Spherical Fourier Neural Operators: Learning Stable Dynamics on the Sphere](https://mlsanity.com/paper/2306.03838/) (2023)
- [Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters](https://mlsanity.com/paper/2403.11549/) (2024)
- [FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2208.05419/) (2022)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Forecasting Global Weather with Graph Neural Networks](https://mlsanity.com/paper/2202.07575/) (2022)
- [Scaling transformer neural networks for skillful and reliable medium-range weather forecasting](https://mlsanity.com/paper/2312.03876/) (2023)
- [Lifelong Language Pretraining with Distribution-Specialized Experts](https://mlsanity.com/paper/2305.12281/) (2023)
- [DiffCast: A Unified Framework via Residual Diffusion for Precipitation Nowcasting](https://mlsanity.com/paper/2312.06734/) (2023)
- [Multi-Task Dense Prediction via Mixture of Low-Rank Experts](https://mlsanity.com/paper/2403.17749/) (2024)
- [Resurrecting Old Classes with New Data for Exemplar-Free Continual Learning](https://mlsanity.com/paper/2405.19074/) (2024)
- [Switch Diffusion Transformer: Synergizing Denoising Tasks with Sparse Mixture-of-Experts](https://mlsanity.com/paper/2403.09176/) (2024)
- [Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts](https://mlsanity.com/paper/2312.00968/) (2023)
- [Generalizing Weather Forecast to Fine-grained Temporal Scales via Physics-AI Hybrid Modeling](https://mlsanity.com/paper/2405.13796/) (2024)
- [ECLIPSE: Efficient Continual Learning in Panoptic Segmentation with Visual Prompt Tuning](https://mlsanity.com/paper/2403.20126/) (2024)
- [Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation](https://mlsanity.com/paper/2404.00417/) (2024)
- [Learning Equi-angular Representations for Online Continual Learning](https://mlsanity.com/paper/2404.01628/) (2024)
- [EWMoE: An effective model for global weather forecasting with mixture-of-experts](https://mlsanity.com/paper/2405.06004/) (2024)
- [Adaptive VIO: Deep Visual-Inertial Odometry with Online Continual Learning](https://mlsanity.com/paper/2405.16754/) (2024)
- [Multi-Content Interaction Network for Few-Shot Segmentation](https://mlsanity.com/paper/2303.06304/) (2023)

## Built on by (most cited first, 6 of 6)

- [STCast: Adaptive Boundary Alignment for Global and Regional Weather Forecasting](https://mlsanity.com/paper/2509.25210/) (2025)
- [EMFormer: Efficient Multi-Scale Transformer for Accumulative Context Weather Forecasting](https://mlsanity.com/paper/2602.01194/) (2026)
- [Generative 3D Gaussian Splatting for Arbitrary-ResolutionAtmospheric Downscaling and Forecasting](https://mlsanity.com/paper/2604.07928/) (2026)
- [RealBench: Benchmarking Data-Driven Numerical Weather Forecasting Under Operational Conditions and Extreme Event Challenges](https://mlsanity.com/paper/2605.24945/) (2026)
- [AdaWeather: Adaptively Mixing Probabilistic Weather Forecasts with Logarithmic Regret](https://mlsanity.com/paper/2606.02663/) (2026)
- [FlexCast: Adaptive Weather Forecasting from Arbitrary Field Sets](https://mlsanity.com/paper/2610.05296/) (2026)

## Abstract (arXiv)

This paper presents Variables Adaptive Mixture of Experts (VAMoE), a novel framework for incremental weather forecasting that dynamically adapts to evolving spatiotemporal patterns in real time data. Traditional weather prediction models often struggle with exorbitant computational expenditure and the need to continuously update forecasts as new observations arrive. VAMoE addresses these challenges by leveraging a hybrid architecture of experts, where each expert specializes in capturing distinct subpatterns of atmospheric variables (temperature, humidity, wind speed). Moreover, the proposed method employs a variable adaptive gating mechanism to dynamically select and combine relevant experts based on the input context, enabling efficient knowledge distillation and parameter sharing. This design significantly reduces computational overhead while maintaining high forecast accuracy. Experiments on real world ERA5 dataset demonstrate that VAMoE performs comparable against SoTA models in both short term (1 days) and long term (5 days) forecasting tasks, with only about 25% of trainable parameters and 50% of the initial training data.