# Multi-modal learning for geospatial vegetation forecasting

arXiv: https://arxiv.org/abs/2303.16198 · HTML: https://mlsanity.com/paper/2303.16198/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 101 references that are in the map.
- First posted on arXiv 28 Mar 2023.

## Builds on (most cited first, 30 of 41)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [FiLM: Visual Reasoning with a General Conditioning Layer](https://mlsanity.com/paper/1709.07871/) (2017)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [Autoformer: Decomposition Transformers with Auto-Correlation for Long-Term Series Forecasting](https://mlsanity.com/paper/2106.13008/) (2021)
- [Group Normalization](https://mlsanity.com/paper/1803.08494/) (2018)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Are Transformers Effective for Time Series Forecasting?](https://mlsanity.com/paper/2205.13504/) (2022)
- [Relational inductive biases, deep learning, and graph networks](https://mlsanity.com/paper/1806.01261/) (2018)
- [Perceiver: General Perception with Iterative Attention](https://mlsanity.com/paper/2103.03206/) (2021)
- [FourCastNet: A Global Data-driven High-resolution Weather Model using Adaptive Fourier Neural Operators](https://mlsanity.com/paper/2202.11214/) (2022)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [Empirical Evaluation of Rectified Activations in Convolutional Network](https://mlsanity.com/paper/1505.00853/) (2015)
- [SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery](https://mlsanity.com/paper/2207.08051/) (2022)
- [ClimaX: A foundation model for weather and climate](https://mlsanity.com/paper/2301.10343/) (2023)
- [Skillful Precipitation Nowcasting using Deep Generative Models of Radar](https://mlsanity.com/paper/2104.00954/) (2021)
- [Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning](https://mlsanity.com/paper/2212.14532/) (2022)
- [WeatherBench: A benchmark dataset for data-driven weather forecasting](https://mlsanity.com/paper/2002.00469/) (2020)
- [SimVP: Simpler yet Better Video Prediction](https://mlsanity.com/paper/2206.05099/) (2022)
- [Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model](https://mlsanity.com/paper/1706.03458/) (2017)
- [PredRNN: A Recurrent Neural Network for Spatiotemporal Predictive Learning](https://mlsanity.com/paper/2103.09504/) (2021)
- [Stochastic Adversarial Video Prediction](https://mlsanity.com/paper/1804.01523/) (2018)
- [Earthformer: Exploring Space-Time Transformers for Earth System Forecasting](https://mlsanity.com/paper/2207.05833/) (2022)
- [MaskViT: Masked Visual Pre-Training for Video Prediction](https://mlsanity.com/paper/2206.11894/) (2022)
- [Towards Geospatial Foundation Models via Continual Pretraining](https://mlsanity.com/paper/2302.04476/) (2023)

## Built on by (most cited first, 16 of 16)

- [REO-VLM: Transforming VLM to Meet Regression Challenges in Earth Observation](https://mlsanity.com/paper/2412.16583/) (2024)
- [Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery](https://mlsanity.com/paper/2507.13385/) (2025)
- [VegeDiff: Latent Diffusion Model for Geospatial Vegetation Forecasting](https://mlsanity.com/paper/2407.12592/) (2024)
- [Exact: Exploring Space-Time Perceptive Clues for Weakly Supervised Satellite Image Time Series Semantic Segmentation](https://mlsanity.com/paper/2412.03968/) (2024)
- [Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach](https://mlsanity.com/paper/2505.17637/) (2025)
- [DeepExtremeCubes: Integrating Earth system spatio-temporal data for impact assessment of climate extremes](https://mlsanity.com/paper/2406.18179/) (2024)
- [Probabilistic NDVI Forecasting from Sparse Satellite Time Series and Weather Covariates](https://mlsanity.com/paper/2602.17683/) (2026)
- [Explainable Earth Surface Forecasting under Extreme Events](https://mlsanity.com/paper/2410.01770/) (2024)
- [UniTS: Unified Spatio-Temporal Generative Model for Remote Sensing](https://mlsanity.com/paper/2512.04461/) (2025)
- [Identifying Spatio-Temporal Drivers of Extreme Events](https://mlsanity.com/paper/2410.24075/) (2024)
- [Towards Unified Vision Language Models for Forest Ecological Analysis in Earth Observation](https://mlsanity.com/paper/2511.16853/) (2025)
- [TEA: Temporal Adaptive Satellite Image Semantic Segmentation](https://mlsanity.com/paper/2601.04956/) (2026)
- [Long-Term Probabilistic Forecast of Vegetation Conditions Using Climate Attributes in the Four Corners Region](https://mlsanity.com/paper/2601.16347/) (2026)
- [GeoRoPE: Ground-Aware Rotary Adaptation for Remote Sensing Foundation Models](https://mlsanity.com/paper/2606.14760/) (2026)
- [HLS-GPT: A Generative Pretrained Transformer (GPT) for Continental-Scale NASA Harmonized Landsat and Sentinel-2 (HLS) Reflectance Reconstruction Across All Bands on Arbitrary Dates](https://mlsanity.com/paper/2606.18115/) (2026)
- [Learning to Forecast Crop Growth from Earth Observation Data](https://mlsanity.com/paper/2608.14281/) (2026)

## Abstract (arXiv)

The innovative application of precise geospatial vegetation forecasting holds immense potential across diverse sectors, including agriculture, forestry, humanitarian aid, and carbon accounting. To leverage the vast availability of satellite imagery for this task, various works have applied deep neural networks for predicting multispectral images in photorealistic quality. However, the important area of vegetation dynamics has not been thoroughly explored. Our study breaks new ground by introducing GreenEarthNet, the first dataset specifically designed for high-resolution vegetation forecasting, and Contextformer, a novel deep learning approach for predicting vegetation greenness from Sentinel 2 satellite images with fine resolution across Europe. Our multi-modal transformer model Contextformer leverages spatial context through a vision backbone and predicts the temporal dynamics on local context patches incorporating meteorological time series in a parameter-efficient manner. The GreenEarthNet dataset features a learned cloud mask and an appropriate evaluation scheme for vegetation modeling. It also maintains compatibility with the existing satellite imagery forecasting dataset EarthNet2021, enabling cross-dataset model comparisons. Our extensive qualitative and quantitative analyses reveal that our methods outperform a broad range of baseline techniques. This includes surpassing previous state-of-the-art models on EarthNet2021, as well as adapted models from time series forecasting and video prediction. To the best of our knowledge, this work presents the first models for continental-scale vegetation modeling at fine resolution able to capture anomalies beyond the seasonal cycle, thereby paving the way for predicting vegetation health and behaviour in response to climate variability and extremes.