# Crop mapping from image time series: deep learning with multi-scale label hierarchies

arXiv: https://arxiv.org/abs/2102.08820 · HTML: https://mlsanity.com/paper/2102.08820/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 82 references that are in the map.
- First posted on arXiv 17 Feb 2021.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting](https://mlsanity.com/paper/1506.04214/) (2015)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Learning to Reweight Examples for Robust Deep Learning](https://mlsanity.com/paper/1803.09050/) (2018)
- [Cost Sensitive Learning of Deep Feature Representations from Imbalanced Data](https://mlsanity.com/paper/1508.03422/) (2015)
- [Striking the Right Balance with Uncertainty](https://mlsanity.com/paper/1901.07590/) (2019)
- [Temporal Convolutional Neural Network for the Classification of Satellite Image Time Series](https://mlsanity.com/paper/1811.10166/) (2018)
- [Self-attention for raw optical Satellite Time Series Classification](https://mlsanity.com/paper/1910.10536/) (2019)
- [Satellite Image Time Series Classification with Pixel-Set Encoders and Temporal Self-Attention](https://mlsanity.com/paper/1911.07757/) (2019)
- [Imbalanced Deep Learning by Minority Class Incremental Rectification](https://mlsanity.com/paper/1804.10851/) (2018)
- [Multi-Temporal Land Cover Classification with Sequential Recurrent Encoders](https://mlsanity.com/paper/1802.02080/) (2018)
- [VideoLSTM Convolves, Attends and Flows for Action Recognition](https://mlsanity.com/paper/1607.01794/) (2016)
- [Convolutional Tensor-Train LSTM for Spatio-temporal Learning](https://mlsanity.com/paper/2002.09131/) (2020)
- [Hierarchical Text Classification with Reinforced Label Assignment](https://mlsanity.com/paper/1908.10419/) (2019)
- [B-CNN: Branch Convolutional Neural Network for Hierarchical Classification](https://mlsanity.com/paper/1709.09890/) (2017)
- [Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning](https://mlsanity.com/paper/1802.05800/) (2018)
- [Meta-Learning for Few-Shot Land Cover Classification](https://mlsanity.com/paper/2004.13390/) (2020)
- [Convolutional Gated Recurrent Networks for Video Segmentation](https://mlsanity.com/paper/1611.05435/) (2016)
- [Time-Space tradeoff in deep learning models for crop classification on satellite multi-spectral image time series](https://mlsanity.com/paper/1901.10503/) (2019)
- [Gating Revisited: Deep Multi-layer RNNs That Can Be Trained](https://mlsanity.com/paper/1911.11033/) (2019)
- [Convolutional LSTMs for Cloud-Robust Segmentation of Remote Sensing Imagery](https://mlsanity.com/paper/1811.02471/) (2018)
- [Combined convolutional and recurrent neural networks for hierarchical classification of images](https://mlsanity.com/paper/1809.09574/) (2018)
- [Learning with Hierarchical Complement Objective](https://mlsanity.com/paper/1911.07257/) (2019)

## Built on by (most cited first, 30 of 42)

- [Panoptic Segmentation of Satellite Image Time Series with Convolutional Temporal Attention Networks](https://mlsanity.com/paper/2107.07933/) (2021)
- [TorchGeo: Deep Learning With Geospatial Data](https://mlsanity.com/paper/2111.08872/) (2021)
- [EarthNets: Empowering AI in Earth Observation](https://mlsanity.com/paper/2210.04936/) (2022)
- [UnCRtainTS: Uncertainty Quantification for Cloud Removal in Optical Satellite Time Series](https://mlsanity.com/paper/2304.05464/) (2023)
- [Artificial intelligence to advance Earth observation: : A review of models, recent trends, and pathways forward](https://mlsanity.com/paper/2305.08413/) (2023)
- [A Sentinel-2 multi-year, multi-country benchmark dataset for crop classification and segmentation with deep learning](https://mlsanity.com/paper/2204.00951/) (2022)
- [EuroCrops: A Pan-European Dataset for Time Series Crop Type Classification](https://mlsanity.com/paper/2106.08151/) (2021)
- [Deep Learning for Satellite Image Time Series Analysis: A Review](https://mlsanity.com/paper/2404.03936/) (2024)
- [EuroCropsML: A Time Series Benchmark Dataset For Few-Shot Crop Type Classification](https://mlsanity.com/paper/2407.17458/) (2024)
- [Taxonomy-Aware Evaluation of Vision-Language Models](https://mlsanity.com/paper/2504.05457/) (2025)
- [Country-wide Retrieval of Forest Structure From Optical and SAR Satellite Imagery With Deep Ensembles](https://mlsanity.com/paper/2111.13154/) (2021)
- [AgriFM: A Multi-source Temporal Remote Sensing Foundation Model for Agriculture Mapping](https://mlsanity.com/paper/2505.21357/) (2025)
- [Boosting Crop Classification by Hierarchically Fusing Satellite, Rotational, and Contextual Data](https://mlsanity.com/paper/2305.12011/) (2023)
- [TIML: Task-Informed Meta-Learning for Agriculture](https://mlsanity.com/paper/2202.02124/) (2022)
- [DISCount: Counting in Large Image Collections with Detector-Based Importance Sampling](https://mlsanity.com/paper/2306.03151/) (2023)
- [FLAIR-HUB: Large-scale Multimodal Dataset for Land Cover and Crop Mapping](https://mlsanity.com/paper/2506.07080/) (2025)
- [Digital Taxonomist: Identifying Plant Species in Community Scientists' Photographs](https://mlsanity.com/paper/2106.03774/) (2021)
- [A systematic review of the use of Deep Learning in Satellite Imagery for Agriculture](https://mlsanity.com/paper/2210.01272/) (2022)
- [Towards Space-to-Ground Data Availability for Agriculture Monitoring](https://mlsanity.com/paper/2205.07721/) (2022)
- [SwissCrop25: A National Multi-Year Benchmark for Operational Crop Mapping](https://mlsanity.com/paper/2608.09497/) (2026)
- [Tree semantic segmentation from aerial image time series](https://mlsanity.com/paper/2407.13102/) (2024)
- [Location Is All You Need: Continuous Spatiotemporal Neural Representations of Earth Observation Data](https://mlsanity.com/paper/2604.07092/) (2026)
- [Using Machine Learning to generate an open-access cropland map from satellite images time series in the Indian Himalayan Region](https://mlsanity.com/paper/2203.14673/) (2022)
- [Satellite-based high-resolution maps of cocoa planted area for Cote d'Ivoire and Ghana](https://mlsanity.com/paper/2206.06119/) (2022)
- [SepHRNet: Generating High-Resolution Crop Maps from Remote Sensing imagery using HRNet with Separable Convolution](https://mlsanity.com/paper/2307.05700/) (2023)
- [Benchmarking for Practice: Few-Shot Time-Series Crop-Type Classification on the EuroCropsML Dataset](https://mlsanity.com/paper/2504.11022/) (2025)
- [From Time-series Generation, Model Selection to Transfer Learning: A Comparative Review of Pixel-wise Approaches for Large-scale Crop Mapping](https://mlsanity.com/paper/2507.12590/) (2025)
- [Habitat classification from satellite observations with sparse annotations](https://mlsanity.com/paper/2209.12995/) (2022)
- [Astronomical image time series classification using CONVolutional attENTION (ConvEntion)](https://mlsanity.com/paper/2304.01236/) (2023)
- [Harnessing Administrative Data Inventories to Create a Reliable Transnational Reference Database for Crop Type Monitoring](https://mlsanity.com/paper/2310.06393/) (2023)

## Abstract (arXiv)

The aim of this paper is to map agricultural crops by classifying satellite image time series. Domain experts in agriculture work with crop type labels that are organised in a hierarchical tree structure, where coarse classes (like orchards) are subdivided into finer ones (like apples, pears, vines, etc.). We develop a crop classification method that exploits this expert knowledge and significantly improves the mapping of rare crop types. The three-level label hierarchy is encoded in a convolutional, recurrent neural network (convRNN), such that for each pixel the model predicts three labels at different level of granularity. This end-to-end trainable, hierarchical network architecture allows the model to learn joint feature representations of rare classes (e.g., apples, pears) at a coarser level (e.g., orchard), thereby boosting classification performance at the fine-grained level. Additionally, labelling at different granularity also makes it possible to adjust the output according to the classification scores; as coarser labels with high confidence are sometimes more useful for agricultural practice than fine-grained but very uncertain labels. We validate the proposed method on a new, large dataset that we make public. ZueriCrop covers an area of 50 km x 48 km in the Swiss cantons of Zurich and Thurgau with a total of 116'000 individual fields spanning 48 crop classes, and 28,000 (multi-temporal) image patches from Sentinel-2. We compare our proposed hierarchical convRNN model with several baselines, including methods designed for imbalanced class distributions. The hierarchical approach performs superior by at least 9.9 percentage points in F1-score.