# Zero-shot Hierarchical Plant Segmentation via Foundation Segmentation Models and Text-to-image Attention

arXiv: https://arxiv.org/abs/2509.09116 · HTML: https://mlsanity.com/paper/2509.09116/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 11 Sep 2025.

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Show, Attend and Tell: Neural Image Caption Generation with Visual Attention](https://mlsanity.com/paper/1502.03044/) (2015)
- [Grounded SAM: Assembling Open-World Models for Diverse Visual Tasks](https://mlsanity.com/paper/2401.14159/) (2024)
- [Panoptic Segmentation](https://mlsanity.com/paper/1801.00868/) (2018)
- [Open-Vocabulary Semantic Segmentation with Mask-adapted CLIP](https://mlsanity.com/paper/2210.04150/) (2022)
- [Segment Everything Everywhere All at Once](https://mlsanity.com/paper/2304.06718/) (2023)
- [Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactively](https://mlsanity.com/paper/2401.02955/) (2024)
- [Hierarchical Open-vocabulary Universal Image Segmentation](https://mlsanity.com/paper/2307.00764/) (2023)
- [PhenoBench -- A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain](https://mlsanity.com/paper/2306.04557/) (2023)
- [Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts](https://mlsanity.com/paper/2404.00741/) (2024)
- [Recurrent Instance Segmentation](https://mlsanity.com/paper/1511.08250/) (2015)
- [Hierarchical Approach for Joint Semantic, Plant Instance, and Leaf Instance Segmentation in the Agricultural Domain](https://mlsanity.com/paper/2210.07879/) (2022)
- [GrowliFlower: An image time series dataset for GROWth analysis of cauLIFLOWER](https://mlsanity.com/paper/2204.00294/) (2022)
- [Leaf Only SAM: A Segment Anything Pipeline for Zero-Shot Automated Leaf Segmentation](https://mlsanity.com/paper/2305.09418/) (2023)
- [Virtual Temporal Samples for Recurrent Neural Networks: applied to semantic segmentation in agriculture](https://mlsanity.com/paper/2106.10118/) (2021)
- [Segment Anything for comprehensive analysis of grapevine cluster architecture and berry properties](https://mlsanity.com/paper/2403.12935/) (2024)

## Built on by (most cited first, 3 of 3)

- [GrowFields: Compositional 4D Neural Fields for Topology-Changing Plant Growth](https://mlsanity.com/paper/2607.03330/) (2026)
- [Text-conditioned Segmentation for Tomato Phenotyping via Procedural Synthetic Data](https://mlsanity.com/paper/2607.18576/) (2026)
- [A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds](https://mlsanity.com/paper/2610.03468/) (2026)

## Abstract (arXiv)

Foundation segmentation models achieve reasonable leaf instance extraction from top-view crop images without training (i.e., zero-shot). However, segmenting entire plant individuals with each consisting of multiple overlapping leaves remains challenging. This problem is referred to as a hierarchical segmentation task, typically requiring annotated training datasets, which are often species-specific and require notable human labor. To address this, we introduce ZeroPlantSeg, a zero-shot segmentation for rosette-shaped plant individuals from top-view images. We integrate a foundation segmentation model, extracting leaf instances, and a vision-language model, reasoning about plants' structures to extract plant individuals without additional training. Evaluations on datasets with multiple plant species, growth stages, and shooting environments demonstrate that our method surpasses existing zero-shot methods and achieves better cross-domain performance than supervised methods. Implementations are available at https://github.com/JunhaoXing/ZeroPlantSeg.