# An Open and Large-Scale Dataset for Multi-Modal Climate Change-aware Crop Yield Predictions

arXiv: https://arxiv.org/abs/2406.06081 · HTML: https://mlsanity.com/paper/2406.06081/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 69 references that are in the map.
- First posted on arXiv 10 Jun 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [Contrast Everything: A Hierarchical Contrastive Framework for Medical Time-Series](https://mlsanity.com/paper/2310.14017/) (2023)
- [A Study of the Attention Abnormality in Trojaned BERTs](https://mlsanity.com/paper/2205.08305/) (2022)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [Open High-Resolution Satellite Imagery: The WorldStrat Dataset -- With Application to Super-Resolution](https://mlsanity.com/paper/2207.06418/) (2022)
- [A CNN-RNN Framework for Crop Yield Prediction](https://mlsanity.com/paper/1911.09045/) (2019)
- [MMST-ViT: Climate Change-aware Crop Yield Prediction via Multi-Modal Spatial-Temporal Vision Transformer](https://mlsanity.com/paper/2309.09067/) (2023)
- [ReLAX: Reinforcement Learning Agent eXplainer for Arbitrary Predictive Models](https://mlsanity.com/paper/2110.11960/) (2021)
- [Large Language Model (LLM) AI text generation detection based on transformer deep learning algorithm](https://mlsanity.com/paper/2405.06652/) (2024)
- [Simultaneous Corn and Soybean Yield Prediction from Remote Sensing Data Using Deep Transfer Learning](https://mlsanity.com/paper/2012.03129/) (2020)
- [RainNet: A Large-Scale Imagery Dataset and Benchmark for Spatial Precipitation Downscaling](https://mlsanity.com/paper/2012.09700/) (2020)
- [CVLight: Decentralized Learning for Adaptive Traffic Signal Control with Connected Vehicles](https://mlsanity.com/paper/2104.10340/) (2021)
- [A Hierarchical Spatial Transformer for Massive Point Samples in Continuous Space](https://mlsanity.com/paper/2311.04434/) (2023)
- [ST-GIN: An Uncertainty Quantification Approach in Traffic Data Imputation with Spatio-temporal Graph Attention and Bidirectional Recurrent United Neural Networks](https://mlsanity.com/paper/2305.06480/) (2023)
- [Adaptive Transfer Learning of Multi-View Time Series Classification](https://mlsanity.com/paper/1910.07632/) (2019)

## Built on by (most cited first, 11 of 11)

- [A Comprehensive Study on Quantization Techniques for Large Language Models](https://mlsanity.com/paper/2411.02530/) (2024)
- [Towards Robust Vision Transformer via Masked Adaptive Ensemble](https://mlsanity.com/paper/2407.15385/) (2024)
- [AgriPotential: A Novel Multi-Spectral and Multi-Temporal Remote Sensing Dataset for Agricultural Potentials](https://mlsanity.com/paper/2506.11740/) (2025)
- [VITA: Variational Pretraining of Transformers for Climate-Robust Crop Yield Forecasting](https://mlsanity.com/paper/2508.03589/) (2025)
- [YieldSAT: A Multimodal Benchmark Dataset for High-Resolution Crop Yield Prediction](https://mlsanity.com/paper/2604.00940/) (2026)
- [Do Protein Transformers Have Biological Intelligence?](https://mlsanity.com/paper/2506.06701/) (2025)
- [DFYP: A Dynamic Fusion Framework with Spectral Channel Attention and Adaptive Operator learning for Crop Yield Prediction](https://mlsanity.com/paper/2507.05849/) (2025)
- [Out-of-Distribution Generalization in Climate-Aware Yield Prediction with Earth Observation Data](https://mlsanity.com/paper/2510.07350/) (2025)
- [Quantizing Space and Time: Fusing Time Series and Images for Earth Observation](https://mlsanity.com/paper/2510.23118/) (2025)
- [M3R: Localized Rainfall Nowcasting with Meteorology-Informed MultiModal Attention](https://mlsanity.com/paper/2604.15377/) (2026)
- [A Multimodal Dataset for Large Language Model Applications in the Energy Domain](https://mlsanity.com/paper/2607.11459/) (2026)

## Abstract (arXiv)

Precise crop yield predictions are of national importance for ensuring food security and sustainable agricultural practices. While AI-for-science approaches have exhibited promising achievements in solving many scientific problems such as drug discovery, precipitation nowcasting, etc., the development of deep learning models for predicting crop yields is constantly hindered by the lack of an open and large-scale deep learning-ready dataset with multiple modalities to accommodate sufficient information. To remedy this, we introduce the CropNet dataset, the first terabyte-sized, publicly available, and multi-modal dataset specifically targeting climate change-aware crop yield predictions for the contiguous United States (U.S.) continent at the county level. Our CropNet dataset is composed of three modalities of data, i.e., Sentinel-2 Imagery, WRF-HRRR Computed Dataset, and USDA Crop Dataset, for over 2200 U.S. counties spanning 6 years (2017-2022), expected to facilitate researchers in developing versatile deep learning models for timely and precisely predicting crop yields at the county-level, by accounting for the effects of both short-term growing season weather variations and long-term climate change on crop yields. Besides, we develop the CropNet package, offering three types of APIs, for facilitating researchers in downloading the CropNet data on the fly over the time and region of interest, and flexibly building their deep learning models for accurate crop yield predictions. Extensive experiments have been conducted on our CropNet dataset via employing various types of deep learning solutions, with the results validating the general applicability and the efficacy of the CropNet dataset in climate change-aware crop yield predictions.