# FusionSF: Fuse Heterogeneous Modalities in a Vector Quantized Framework for Robust Solar Power Forecasting

arXiv: https://arxiv.org/abs/2402.05823 · HTML: https://mlsanity.com/paper/2402.05823/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 51 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 15 of 15)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Informer: Beyond Efficient Transformer for Long Sequence Time-Series Forecasting](https://mlsanity.com/paper/2012.07436/) (2020)
- [A Time Series is Worth 64 Words: Long-term Forecasting with Transformers](https://mlsanity.com/paper/2211.14730/) (2022)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [FEDformer: Frequency Enhanced Decomposed Transformer for Long-term Series Forecasting](https://mlsanity.com/paper/2201.12740/) (2022)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [One Fits All:Power General Time Series Analysis by Pretrained LM](https://mlsanity.com/paper/2302.11939/) (2023)
- [Less Is More: Fast Multivariate Time Series Forecasting with Light Sampling-oriented MLP Structures](https://mlsanity.com/paper/2207.01186/) (2022)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [Meta-learning framework with applications to zero-shot time-series forecasting](https://mlsanity.com/paper/2002.02887/) (2020)
- [Sky Imager-Based Forecast of Solar Irradiance Using Machine Learning](https://mlsanity.com/paper/2310.17356/) (2023)

## Built on by (most cited first, 6 of 6)

- [Time-MQA: Time Series Multi-Task Question Answering with Context Enhancement](https://mlsanity.com/paper/2503.01875/) (2025)
- [Unraveling Spatio-Temporal Foundation Models via the Pipeline Lens: A Comprehensive Review](https://mlsanity.com/paper/2506.01364/) (2025)
- [Selective Learning for Deep Time Series Forecasting](https://mlsanity.com/paper/2510.25207/) (2025)
- [Collaborative Deterministic-Probabilistic Forecasting for Diverse Spatiotemporal Systems](https://mlsanity.com/paper/2502.11013/) (2025)
- [Integrating Weather Foundation Model and Satellite to Enable Fine-Grained Solar Irradiance Forecasting](https://mlsanity.com/paper/2603.14845/) (2026)
- [UniWind: Day-Ahead Wind Power Forecasting via Physics-Informed State Routing](https://mlsanity.com/paper/2607.01670/) (2026)

## Abstract (arXiv)

Accurate solar power forecasting is crucial to integrate photovoltaic plants into the electric grid, schedule and secure the power grid safety. This problem becomes more demanding for those newly installed solar plants which lack sufficient data. Current research predominantly relies on historical solar power data or numerical weather prediction in a single-modality format, ignoring the complementary information provided in different modalities. In this paper, we propose a multi-modality fusion framework to integrate historical power data, numerical weather prediction, and satellite images, significantly improving forecast performance. We introduce a vector quantized framework that aligns modalities with varying information densities, striking a balance between integrating sufficient information and averting model overfitting. Our framework demonstrates strong zero-shot forecasting capability, which is especially useful for those newly installed plants. Moreover, we collect and release a multi-modal solar power (MMSP) dataset from real-world plants to further promote the research of multi-modal solar forecasting algorithms. Our extensive experiments show that our model not only operates with robustness but also boosts accuracy in both zero-shot forecasting and scenarios rich with training data, surpassing leading models. We have incorporated it into our eForecaster platform and deployed it for more than 300 solar plants with a capacity of over 15GW.