# MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer

arXiv: https://arxiv.org/abs/2309.09067 · HTML: https://mlsanity.com/paper/2309.09067/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 32 of its 59 references that are in the map.
- First posted on arXiv 16 Sep 2023.

## Builds on (most cited first, 30 of 32)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutions](https://mlsanity.com/paper/2102.12122/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training](https://mlsanity.com/paper/2203.12602/) (2022)
- [Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet](https://mlsanity.com/paper/2101.11986/) (2021)
- [Multiscale Vision Transformers](https://mlsanity.com/paper/2104.11227/) (2021)
- [PVT v2: Improved Baselines with Pyramid Vision Transformer](https://mlsanity.com/paper/2106.13797/) (2021)
- [MViTv2: Improved Multiscale Vision Transformers for Classification and Detection](https://mlsanity.com/paper/2112.01526/) (2021)
- [Rethinking Spatial Dimensions of Vision Transformers](https://mlsanity.com/paper/2103.16302/) (2021)
- [Tackling Climate Change with Machine Learning](https://mlsanity.com/paper/1906.05433/) (2019)
- [UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training](https://mlsanity.com/paper/2002.12804/) (2020)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [A CNN-RNN Framework for Crop Yield Prediction](https://mlsanity.com/paper/1911.09045/) (2019)
- [Crop Yield Prediction Using Deep Neural Networks](https://mlsanity.com/paper/1902.02860/) (2019)
- [A GNN-RNN Approach for Harnessing Geospatial and Temporal Information: Application to Crop Yield Prediction](https://mlsanity.com/paper/2111.08900/) (2021)
- [Learning the Distribution of Errors in Stereo Matching for Joint Disparity and Uncertainty Estimation](https://mlsanity.com/paper/2304.00152/) (2023)

## Built on by (most cited first, 29 of 29)

- [Medformer: A Multi-Granularity Patching Transformer for Medical Time-Series Classification](https://mlsanity.com/paper/2405.19363/) (2024)
- [A Comprehensive Study on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2411.02530/) (2024)
- [How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlook](https://mlsanity.com/paper/2503.11835/) (2025)
- [An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictions](https://mlsanity.com/paper/2406.06081/) (2024)
- [Blockchain for Large Language Model Security and Safety: A Holistic Survey](https://mlsanity.com/paper/2407.20181/) (2024)
- [Mapping New Realities: Ground Truth Image Creation with Pix2Pix Image-to-Image Translation](https://mlsanity.com/paper/2404.19265/) (2024)
- [Uncertainty-Aware Crime Prediction With Spatial Temporal Multivariate Graph Neural Networks](https://mlsanity.com/paper/2408.04193/) (2024)
- [YOLO11 and Vision Transformers based 3D Pose Estimation of Immature Green Fruits in Commercial Apple Orchards for Robotic Thinning](https://mlsanity.com/paper/2410.19846/) (2024)
- [Causal Spatio-Temporal Prediction: An Effective and Efficient Multi-Modal Approach](https://mlsanity.com/paper/2505.17637/) (2025)
- [From Twitter to Reasoner: Understand Mobility Travel Modes and Sentiment Using Large Language Models](https://mlsanity.com/paper/2411.02666/) (2024)
- [A Survey on Cloud-Edge-Terminal Collaborative Intelligence in AIoT Networks](https://mlsanity.com/paper/2508.18803/) (2025)
- [From General to Specialized: The Need for Foundational Models in Agriculture](https://mlsanity.com/paper/2507.05390/) (2025)
- [Integrating Spatiotemporal Vision Transformer into Digital Twins for High-Resolution Heat Stress Forecasting in Campus Environments](https://mlsanity.com/paper/2502.09657/) (2025)
- [California Crop Yield Benchmark: Combining Satellite Image, Climate, Evapotranspiration, and Soil Data Layers for County-Level Yield Forecasting of Over 70 Crops](https://mlsanity.com/paper/2506.10228/) (2025)
- [Towards Interpretable Adversarial Examples via Sparse Adversarial Attack](https://mlsanity.com/paper/2506.17250/) (2025)
- [A multi-temporal multi-spectral attention-augmented deep convolution neural network with contrastive learning for crop yield prediction](https://mlsanity.com/paper/2509.15966/) (2025)
- [Long-Tailed Recognition via Information-Preservable Two-Stage Learning](https://mlsanity.com/paper/2510.08836/) (2025)
- [Modeling Time-Lapse Trajectories to Characterize Cranberry Growth](https://mlsanity.com/paper/2510.08901/) (2025)
- [Learning county from pixels: corn yield prediction with attention-weighted multiple instance learning](https://mlsanity.com/paper/2312.01001/) (2023)
- [VN-Net: Vision-Numerical Fusion Graph Convolutional Network for Sparse Spatio-Temporal Meteorological Forecasting](https://mlsanity.com/paper/2404.16037/) (2024)
- [Do Protein Transformers Have Biological Intelligence?](https://mlsanity.com/paper/2506.06701/) (2025)
- [DFYP: A Dynamic Fusion Framework with Spectral Channel Attention and Adaptive Operator learning for Crop Yield Prediction](https://mlsanity.com/paper/2507.05849/) (2025)
- [AI in Agriculture: A Survey of Deep Learning Techniques for Crops, Fisheries and Livestock](https://mlsanity.com/paper/2507.22101/) (2025)
- [Out-of-Distribution Generalization in Climate-Aware Yield Prediction with Earth Observation Data](https://mlsanity.com/paper/2510.07350/) (2025)
- [Quantizing Space and Time: Fusing Time Series and Images for Earth Observation](https://mlsanity.com/paper/2510.23118/) (2025)
- [PollutionNet: A Vision Transformer Framework for Climatological Assessment of NO2 and SO2 Using Satellite-Ground Data Fusion](https://mlsanity.com/paper/2604.03311/) (2026)
- [M3R: Localized Rainfall Nowcasting with Meteorology-Informed MultiModal Attention](https://mlsanity.com/paper/2604.15377/) (2026)
- [Agent-Based Post-Hoc Correction of Agricultural Yield Forecasts](https://mlsanity.com/paper/2605.12375/) (2026)
- [PhenoYieldNet: Learning Crop-Aware Phenological Responses for Multi-Crop Yield Prediction](https://mlsanity.com/paper/2605.23478/) (2026)

## Abstract (arXiv)

Precise crop yield prediction provides valuable information for agricultural planning and decision-making processes. However, timely predicting crop yields remains challenging as crop growth is sensitive to growing season weather variation and climate change. In this work, we develop a deep learning-based solution, namely Multi-Modal Spatial-Temporal Vision Transformer (MMST-ViT), for predicting crop yields at the county level across the United States, by considering the effects of short-term meteorological variations during the growing season and the long-term climate change on crops. Specifically, our MMST-ViT consists of a Multi-Modal Transformer, a Spatial Transformer, and a Temporal Transformer. The Multi-Modal Transformer leverages both visual remote sensing data and short-term meteorological data for modeling the effect of growing season weather variations on crop growth. The Spatial Transformer learns the high-resolution spatial dependency among counties for accurate agricultural tracking. The Temporal Transformer captures the long-range temporal dependency for learning the impact of long-term climate change on crops. Meanwhile, we also devise a novel multi-modal contrastive learning technique to pre-train our model without extensive human supervision. Hence, our MMST-ViT captures the impacts of both short-term weather variations and long-term climate change on crops by leveraging both satellite images and meteorological data. We have conducted extensive experiments on over 200 counties in the United States, with the experimental results exhibiting that our MMST-ViT outperforms its counterparts under three performance metrics of interest.