# Utilizing Strategic Pre-training to Reduce Overfitting: Baguan -- A Pre-trained Weather Forecasting Model

arXiv: https://arxiv.org/abs/2505.13873 · HTML: https://mlsanity.com/paper/2505.13873/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 32 references that are in the map.
- First posted on arXiv 20 May 2025.

## Builds on (most cited first, 15 of 15)

- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [SimMIM: A Simple Framework for Masked Image Modeling](https://mlsanity.com/paper/2111.09886/) (2021)
- [FuXi: A cascade machine learning forecasting system for 15-day global weather forecast](https://mlsanity.com/paper/2306.12873/) (2023)
- [FourCastNet: Accelerating Global High-Resolution Weather Forecasting using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2208.05419/) (2022)
- [FengWu: Pushing the Skillful Global Medium-range Weather Forecast beyond 10 Days Lead](https://mlsanity.com/paper/2304.02948/) (2023)
- [Scaling transformer neural networks for skillful and reliable medium-range weather forecasting](https://mlsanity.com/paper/2312.03876/) (2023)
- [Siamese Masked Autoencoders](https://mlsanity.com/paper/2305.14344/) (2023)
- [ChaosBench: A Multi-Channel, Physics-Based Benchmark for Subseasonal-to-Seasonal Climate Prediction](https://mlsanity.com/paper/2402.00712/) (2024)
- [Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priors](https://mlsanity.com/paper/2310.02980/) (2023)
- [W-MAE: Pre-trained weather model with masked autoencoder for multi-variable weather forecasting](https://mlsanity.com/paper/2304.08754/) (2023)
- [Deep Learning and Foundation Models for Weather Prediction: A Survey](https://mlsanity.com/paper/2501.06907/) (2025)
- [FuXi-2.0: Advancing machine learning weather forecasting model for practical applications](https://mlsanity.com/paper/2409.07188/) (2024)
- [Maximizing the Impact of Deep Learning on Subseasonal-to-Seasonal Climate Forecasting: The Essential Role of Optimization](https://mlsanity.com/paper/2411.16728/) (2024)

## Built on by (most cited first, 7 of 7)

- [Bridging Past and Future: Distribution-Aware Alignment for Time Series Forecasting](https://mlsanity.com/paper/2509.14181/) (2025)
- [Learning to Advect: A Neural Semi-Lagrangian Architecture for Weather Forecasting](https://mlsanity.com/paper/2601.21151/) (2026)
- [Symmetric Composition of Anisotropic Operators for Global Subseasonal-to-Seasonal Climate Forecasting](https://mlsanity.com/paper/2602.15040/) (2026)
- [Integrating Weather Foundation Model and Satellite to Enable Fine-Grained Solar Irradiance Forecasting](https://mlsanity.com/paper/2603.14845/) (2026)
- [Skillful Kilometer-Scale Regional Weather Forecasting via Global and Regional Coupling](https://mlsanity.com/paper/2603.28173/) (2026)
- [Pushing the Limits of High-Resolution Weather Forecasting through Data Scaling](https://mlsanity.com/paper/2608.14652/) (2026)
- [S2S-JEPA: Predicting the Predictable at Subseasonal-to-Seasonal Timescales](https://mlsanity.com/paper/2610.03106/) (2026)

## Abstract (arXiv)

Weather forecasting has long posed a significant challenge for humanity. While recent AI-based models have surpassed traditional numerical weather prediction (NWP) methods in global forecasting tasks, overfitting remains a critical issue due to the limited availability of real-world weather data spanning only a few decades. Unlike fields like computer vision or natural language processing, where data abundance can mitigate overfitting, weather forecasting demands innovative strategies to address this challenge with existing data. In this paper, we explore pre-training methods for weather forecasting, finding that selecting an appropriately challenging pre-training task introduces locality bias, effectively mitigating overfitting and enhancing performance. We introduce Baguan, a novel data-driven model for medium-range weather forecasting, built on a Siamese Autoencoder pre-trained in a self-supervised manner and fine-tuned for different lead times. Experimental results show that Baguan outperforms traditional methods, delivering more accurate forecasts. Additionally, the pre-trained Baguan demonstrates robust overfitting control and excels in downstream tasks, such as subseasonal-to-seasonal (S2S) modeling and regional forecasting, after fine-tuning.