# Leveraging Vision Language Models for Specialized Agricultural Tasks

arXiv: https://arxiv.org/abs/2407.19617 · HTML: https://mlsanity.com/paper/2407.19617/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Agricultural vision-language models](https://mlsanity.com/mllm/agricultural-vision-language-models/)

## Builds on (most cited first, 13 of 13)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [Using Deep Learning for Image-Based Plant Disease Detection](https://mlsanity.com/paper/1604.03169/) (2016)
- [MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding](https://mlsanity.com/paper/2406.09411/) (2024)
- [Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluation](https://mlsanity.com/paper/2305.16938/) (2023)
- [VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use](https://mlsanity.com/paper/2308.06595/) (2023)
- [DeepWeeds: A Multiclass Weed Species Image Dataset for Deep Learning](https://mlsanity.com/paper/1810.05726/) (2018)
- [PlantDoc: A Dataset for Visual Plant Disease Detection](https://mlsanity.com/paper/1911.10317/) (2019)
- [Many-Shot In-Context Learning in Multimodal Foundation Models](https://mlsanity.com/paper/2405.09798/) (2024)
- [MangoLeafBD: A Comprehensive Image Dataset to Classify Diseased and Healthy Mango Leaves](https://mlsanity.com/paper/2209.02377/) (2022)

## Built on by (most cited first, 25 of 25)

- [A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers](https://mlsanity.com/paper/2508.21148/) (2025)
- [A Survey of Multimodal Hallucination Evaluation and Detection](https://mlsanity.com/paper/2507.19024/) (2025)
- [AgriCLIP: Adapting CLIP for Agriculture and Livestock via Domain-Specialized Cross-Model Alignment](https://mlsanity.com/paper/2410.01407/) (2024)
- [AgriGPT-VL: Agricultural Vision-Language Understanding Suite](https://mlsanity.com/paper/2510.04002/) (2025)
- [PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotyping](https://mlsanity.com/paper/2504.19818/) (2025)
- [AgriRegion: Region-Aware Retrieval for High-Fidelity Agricultural Advice](https://mlsanity.com/paper/2512.10114/) (2025)
- [Towards Large Reasoning Models for Agriculture](https://mlsanity.com/paper/2505.19259/) (2025)
- [WisWheat: A Three-Tiered Vision-Language Dataset for Wheat Management](https://mlsanity.com/paper/2506.06084/) (2025)
- [Domain Specific Benchmarks for Evaluating Multimodal Large Language Models](https://mlsanity.com/paper/2506.12958/) (2025)
- [Advances in LLMs with Focus on Reasoning, Adaptability, Efficiency and Ethics](https://mlsanity.com/paper/2506.12365/) (2025)
- [Are vision-language models ready to zero-shot replace supervised classification models in agriculture?](https://mlsanity.com/paper/2512.15977/) (2025)
- [CPJ: Explainable Agricultural Pest Diagnosis via Caption-Prompt-Judge with LLM-Judged Refinement](https://mlsanity.com/paper/2512.24947/) (2025)
- [Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentation](https://mlsanity.com/paper/2602.23677/) (2026)
- [Multi-label Instance-level Generalised Visual Grounding in Agriculture](https://mlsanity.com/paper/2603.06699/) (2026)
- [LinMU: Multimodal Understanding Made Linear](https://mlsanity.com/paper/2601.01322/) (2026)
- [Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning](https://mlsanity.com/paper/2603.08930/) (2026)
- [AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning](https://mlsanity.com/paper/2603.14342/) (2026)
- [Agri-CPJ: A Training-Free Explainable Framework for Agricultural Pest Diagnosis Using Caption-Prompt-Judge and LLM-as-a-Judge](https://mlsanity.com/paper/2604.23701/) (2026)
- [Benchmarking Vision-Language Models for Microscopic Plant Image Understanding](https://mlsanity.com/paper/2606.22497/) (2026)
- [PhenoNEST: A Neuro-Symbolic Framework for Ontology-Aware Multimodal Plant Phenotyping and Trait Discovery](https://mlsanity.com/paper/2607.03245/) (2026)
- [Vision-language models know more about agriculture than they show and rubric-grounded verifications close the gap](https://mlsanity.com/paper/2609.09417/) (2026)
- [From Vision to Harvest: Benchmarking Vision-Language Models for Multi-Arm Robotic Fruit Harvesting](https://mlsanity.com/paper/2609.13606/) (2026)
- [A Multi-Modal Generative Model for Tomato Disease Leaves Understanding](https://mlsanity.com/paper/2609.19555/) (2026)
- [Recent Advances in Agentic Agri-Robotic Phenotyping: A Perspective Review from Fragmented Multimodal Sensing to Unified PhenoAgent Intelligence](https://mlsanity.com/paper/2609.34567/) (2026)
- [A Vision-Language Model (VLM)-based Pipeline for End-to-End Procedural Modeling of Field-Grown Maize from Point Clouds](https://mlsanity.com/paper/2610.03468/) (2026)

## Abstract (arXiv)

As Vision Language Models (VLMs) become increasingly accessible to farmers and agricultural experts, there is a growing need to evaluate their potential in specialized tasks. We present AgEval, a comprehensive benchmark for assessing VLMs' capabilities in plant stress phenotyping, offering a solution to the challenge of limited annotated data in agriculture. Our study explores how general-purpose VLMs can be leveraged for domain-specific tasks with only a few annotated examples, providing insights into their behavior and adaptability. AgEval encompasses 12 diverse plant stress phenotyping tasks, evaluating zero-shot and few-shot in-context learning performance of state-of-the-art models including Claude, GPT, Gemini, and LLaVA. Our results demonstrate VLMs' rapid adaptability to specialized tasks, with the best-performing model showing an increase in F1 scores from 46.24% to 73.37% in 8-shot identification. To quantify performance disparities across classes, we introduce metrics such as the coefficient of variation (CV), revealing that VLMs' training impacts classes differently, with CV ranging from 26.02% to 58.03%. We also find that strategic example selection enhances model reliability, with exact category examples improving F1 scores by 15.38% on average. AgEval establishes a framework for assessing VLMs in agricultural applications, offering valuable benchmarks for future evaluations. Our findings suggest that VLMs, with minimal few-shot examples, show promise as a viable alternative to traditional specialized models in plant stress phenotyping, while also highlighting areas for further refinement. Results and benchmark details are available at: https://github.com/arbab-ml/AgEval