# CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis

arXiv: https://arxiv.org/abs/2605.03259 · HTML: https://mlsanity.com/paper/2605.03259/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 22 of its 52 references that are in the map.
- First posted on arXiv 5 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Agricultural vision-language models](https://mlsanity.com/mllm/agricultural-vision-language-models/)

## Builds on (most cited first, 22 of 22)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Rich feature hierarchies for accurate object detection and semantic segmentation](https://mlsanity.com/paper/1311.2524/) (2013)
- [YOLOv9: Learning What You Want to Learn Using Programmable Gradient Information](https://mlsanity.com/paper/2402.13616/) (2024)
- [Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model](https://mlsanity.com/paper/2203.14940/) (2022)
- [BioCLIP: A Vision Foundation Model for the Tree of Life](https://mlsanity.com/paper/2311.18803/) (2023)
- [A Closer Look at the Explainability of Contrastive Language-Image Pre-training](https://mlsanity.com/paper/2304.05653/) (2023)
- [Faster ILOD: Incremental Learning for Object Detectors based on Faster RCNN](https://mlsanity.com/paper/2003.03901/) (2020)
- [BioTrove: A Large Curated Image Dataset Enabling AI for Biodiversity](https://mlsanity.com/paper/2406.17720/) (2024)
- [AgroBench: Vision-Language Model Benchmark in Agriculture](https://mlsanity.com/paper/2507.20519/) (2025)
- [Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline](https://mlsanity.com/paper/2408.03120/) (2024)
- [GridCLIP: One-Stage Object Detection by Grid-Level CLIP Representation Learning](https://mlsanity.com/paper/2303.09252/) (2023)
- [U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation](https://mlsanity.com/paper/2405.15365/) (2024)
- [AgriCLIP: Adapting CLIP for Agriculture and Livestock via Domain-Specialized Cross-Model Alignment](https://mlsanity.com/paper/2410.01407/) (2024)
- [Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild](https://mlsanity.com/paper/2408.14723/) (2024)
- [Self-Supervised Leaf Segmentation under Complex Lighting Conditions](https://mlsanity.com/paper/2203.15943/) (2022)
- [Zero-Shot Semantic Segmentation via Spatial and Multi-Scale Aware Visual Class Embedding](https://mlsanity.com/paper/2111.15181/) (2021)
- [Enhancing Novel Object Detection via Cooperative Foundational Models](https://mlsanity.com/paper/2311.12068/) (2023)
- [BSDP: Brain-inspired Streaming Dual-level Perturbations for Online Open World Object Detection](https://mlsanity.com/paper/2403.02637/) (2024)

## Abstract (arXiv)

High-throughput plant phenotyping, the quantitative measurement of observable plant traits, is critical for modern breeding but remains constrained by a "phenotyping bottleneck," where manual data collection is labor-intensive and prone to observer bias. Conventional closed-set computer vision systems fail to address this challenge, as they require extensive species-specific annotation and lack the flexibility to handle diverse breeding populations. To bridge this gap, we present CropVLM, a Vision-Language Model (VLM) adapted for the agricultural domain via Domain-Specific Semantic Alignment (DSSA). Trained on 52,987 manually selected image-caption pairs covering 37 species in natural field conditions, CropVLM effectively maps agronomic terminology to fine-grained visual features. We further introduce the Hybrid Open-Set Localization Network (HOS-Net), an architecture that integrates CropVLM to enable the detection of novel crops solely from natural language descriptions without retraining. By eliminating the reliance on species-specific training data, CropVLM provides a scalable solution for high-throughput phenotyping, accelerating genetic gain and facilitating large-scale biodiversity research essential for sustainable agriculture. The trained model weights and complete pipeline implementation are publicly available at: [https://github.com/boudiafA/CropVLM](https://github.com/boudiafA/CropVLM). In comprehensive evaluations, CropVLM achieves 72.51% zero-shot classification accuracy, outperforming seven CLIP-style baselines. Our detection pipeline demonstrates superior zero-shot generalization to novel species, achieving 49.17 AP50 on our CVTCropDet benchmark and 50.73 AP50 on tropical fruit species, compared to 34.89 and 48.58 for the next-best method, respectively.