# GMT: Guided Mask Transformer for Leaf Instance Segmentation

arXiv: https://arxiv.org/abs/2406.17109 · HTML: https://mlsanity.com/paper/2406.17109/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 55 references that are in the map.
- First posted on arXiv 24 Jun 2024.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Deformable DETR: Deformable Transformers for End-to-End Object Detection](https://mlsanity.com/paper/2010.04159/) (2020)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [Masked-attention Mask Transformer for Universal Image Segmentation](https://mlsanity.com/paper/2112.01527/) (2021)
- [V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentation](https://mlsanity.com/paper/1606.04797/) (2016)
- [Per-Pixel Classification is Not All You Need for Semantic Segmentation](https://mlsanity.com/paper/2107.06278/) (2021)
- [Image Segmentation Using Deep Learning: A Survey](https://mlsanity.com/paper/2001.05566/) (2020)
- [Hybrid Task Cascade for Instance Segmentation](https://mlsanity.com/paper/1901.07518/) (2019)
- [OneFormer: One Transformer to Rule Universal Image Segmentation](https://mlsanity.com/paper/2211.06220/) (2022)
- [Semantic Instance Segmentation with a Discriminative Loss Function](https://mlsanity.com/paper/1708.02551/) (2017)
- [A Survey on Instance Segmentation: State of the art](https://mlsanity.com/paper/2007.00047/) (2020)
- [FastInst: A Simple Query-Based Model for Real-Time Instance Segmentation](https://mlsanity.com/paper/2303.08594/) (2023)
- [Deep Leaf Segmentation Using Synthetic Data](https://mlsanity.com/paper/1807.10931/) (2018)
- [Dynamic Focus-aware Positional Queries for Semantic Segmentation](https://mlsanity.com/paper/2204.01244/) (2022)
- [Instance Segmentation of Biological Images Using Harmonic Embeddings](https://mlsanity.com/paper/1904.05257/) (2019)
- [Sparse Object-level Supervision for Instance Segmentation with Pixel Embeddings](https://mlsanity.com/paper/2103.14572/) (2021)
- [PST: Plant segmentation transformer for 3D point clouds of rapeseed plants at the podding stage](https://mlsanity.com/paper/2206.13082/) (2022)
- [Scalable learning for bridging the species gap in image-based plant phenotyping](https://mlsanity.com/paper/2003.10757/) (2020)
- [Object-Guided Instance Segmentation With Auxiliary Feature Refinement for Biological Images](https://mlsanity.com/paper/2106.07159/) (2021)
- [Joint Multi-Leaf Segmentation, Alignment and Tracking from Fluorescence Plant Videos](https://mlsanity.com/paper/1505.00353/) (2015)

## Built on by (most cited first, 2 of 2)

- [PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotyping](https://mlsanity.com/paper/2504.19818/) (2025)
- [GrowFields: Compositional 4D Neural Fields for Topology-Changing Plant Growth](https://mlsanity.com/paper/2607.03330/) (2026)

## Abstract (arXiv)

Leaf instance segmentation is a challenging multi-instance segmentation task, aiming to separate and delineate each leaf in an image of a plant. Accurate segmentation of each leaf is crucial for plant-related applications such as the fine-grained monitoring of plant growth and crop yield estimation. This task is challenging because of the high similarity (in shape and colour), great size variation, and heavy occlusions among leaf instances. Furthermore, the typically small size of annotated leaf datasets makes it more difficult to learn the distinctive features needed for precise segmentation. We hypothesise that the key to overcoming the these challenges lies in the specific spatial patterns of leaf distribution. In this paper, we propose the Guided Mask Transformer (GMT), which leverages and integrates leaf spatial distribution priors into a Transformer-based segmentor. These spatial priors are embedded in a set of guide functions that map leaves at different positions into a more separable embedding space. Our GMT consistently outperforms the state-of-the-art on three public plant datasets. Our code is available at https://github.com/vios-s/gmt-leaf-ins-seg.