# AgroGPT: Efficient Agricultural Vision-Language Model with Expert Tuning

arXiv: https://arxiv.org/abs/2410.08405 · HTML: https://mlsanity.com/paper/2410.08405/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 36 of its 48 references that are in the map.
- First posted on arXiv 10 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Agricultural vision-language models](https://mlsanity.com/mllm/agricultural-vision-language-models/)

## Builds on (most cited first, 30 of 36)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [Video-ChatGPT: Towards Detailed Video Understanding via Large Vision and Language Models](https://mlsanity.com/paper/2306.05424/) (2023)
- [Instruction Tuning with GPT-4](https://mlsanity.com/paper/2304.03277/) (2023)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [Otter: A Multi-Modal Model with In-Context Instruction Tuning](https://mlsanity.com/paper/2305.03726/) (2023)
- [BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs](https://mlsanity.com/paper/2303.00915/) (2023)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [GeoChat: Grounded Large Vision-Language Model for Remote Sensing](https://mlsanity.com/paper/2311.15826/) (2023)
- [LoRA Learns Less and Forgets Less](https://mlsanity.com/paper/2405.09673/) (2024)
- [RSGPT: A Remote Sensing Vision Language Model and Benchmark](https://mlsanity.com/paper/2307.15266/) (2023)
- [Summary of ChatGPT-Related Research and Perspective Towards the Future of Large Language Models](https://mlsanity.com/paper/2304.01852/) (2023)
- [An open access repository of images on plant health to enable the development of mobile disease diagnostics](https://mlsanity.com/paper/1511.08060/) (2015)
- [Foundational Models Defining a New Era in Vision: A Survey and Outlook](https://mlsanity.com/paper/2307.13721/) (2023)
- [PlantDoc: A Dataset for Visual Plant Disease Detection](https://mlsanity.com/paper/1911.10317/) (2019)
- [XrayGPT: Chest Radiographs Summarization using Medical Vision-Language Models](https://mlsanity.com/paper/2306.07971/) (2023)
- [ClimateGPT: Towards AI Synthesizing Interdisciplinary Research on Climate Change](https://mlsanity.com/paper/2401.09646/) (2024)

## Built on by (most cited first, 24 of 24)

- [AgroBench: Vision-Language Model Benchmark in Agriculture](https://mlsanity.com/paper/2507.20519/) (2025)
- [CAMEL-Bench: A Comprehensive Arabic LMM Benchmark](https://mlsanity.com/paper/2410.18976/) (2024)
- [AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark](https://mlsanity.com/paper/2504.10568/) (2025)
- [AgriGPT: a Large Language Model Ecosystem for Agriculture](https://mlsanity.com/paper/2508.08632/) (2025)
- [LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseases](https://mlsanity.com/paper/2602.13662/) (2026)
- [PhenoAssistant: A Conversational Multi-Agent AI System for Automated Plant Phenotyping](https://mlsanity.com/paper/2504.19818/) (2025)
- [AgriGPT-Omni: A Unified Speech-Vision-Text Framework for Multilingual Agricultural Intelligence](https://mlsanity.com/paper/2512.10624/) (2025)
- [WisWheat: A Three-Tiered Vision-Language Dataset for Wheat Management](https://mlsanity.com/paper/2506.06084/) (2025)
- [PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science](https://mlsanity.com/paper/2508.17117/) (2025)
- [AgriChat: A Multimodal Large Language Model for Agriculture Image Understanding](https://mlsanity.com/paper/2603.16934/) (2026)
- [Vision-Language Semantic Grounding for Multi-Domain Crop-Weed Segmentation](https://mlsanity.com/paper/2602.23677/) (2026)
- [Self-Consistency in Vision-Language Models for Precision Agriculture: Multi-Response Consensus for Crop Disease Management](https://mlsanity.com/paper/2507.08024/) (2025)
- [AI in Agriculture: A Survey of Deep Learning Techniques for Crops, Fisheries and Livestock](https://mlsanity.com/paper/2507.22101/) (2025)
- [Agro-Consensus: Semantic Self-Consistency in Vision-Language Models for Crop Disease Management in Developing Countries](https://mlsanity.com/paper/2510.21757/) (2025)
- [Using Vision Language Foundation Models to Generate Plant Simulation Configurations via In-Context Learning](https://mlsanity.com/paper/2603.08930/) (2026)
- [AgriPath: A Systematic Exploration of Architectural Trade-offs for Crop Disease Classification](https://mlsanity.com/paper/2603.13354/) (2026)
- [AgroOmni: A Large-Scale Multi-view Agricultural Dataset for Cross-Scale Multimodal Reasoning](https://mlsanity.com/paper/2603.14342/) (2026)
- [Captioning Daily Activity Images in Early Childhood Education: Benchmark and Algorithm](https://mlsanity.com/paper/2604.01941/) (2026)
- [CultivAgents: Cultivating Relationship-Centered Multi-Agent Systems for Personalized Gardening](https://mlsanity.com/paper/2605.23193/) (2026)
- [Fine-Tuning General-Purpose Large Language Models for Agricultural Applications:A Reproducible Framework and Evaluation Protocol Based on Qwen3-8B](https://mlsanity.com/paper/2606.28992/) (2026)
- [A Multi-Modal Generative Model for Tomato Disease Leaves Understanding](https://mlsanity.com/paper/2609.19555/) (2026)
- [AgriScope: Pixel-Grounded Multimodal Understanding for Agricultural Images](https://mlsanity.com/paper/2609.20325/) (2026)
- [BananaVLM: A Domain-Adapted Vision Language Model for Banana Crop Disease Diagnosis](https://mlsanity.com/paper/2609.25040/) (2026)
- [Uncertainty-Aware Semi-Supervised Framework with Supervised Contrastive Learning for Sugarcane Leaf Disease Detection](https://mlsanity.com/paper/2610.17996/) (2026)

## Abstract (arXiv)

Significant progress has been made in advancing large multimodal conversational models (LMMs), capitalizing on vast repositories of image-text data available online. Despite this progress, these models often encounter substantial domain gaps, hindering their ability to engage in complex conversations across new domains. Recent efforts have aimed to mitigate this issue, albeit relying on domain-specific image-text data to curate instruction-tuning data. However, many domains, such as agriculture, lack such vision-language data. In this work, we propose an approach to construct instruction-tuning data that harnesses vision-only data for the agriculture domain. We utilize diverse agricultural datasets spanning multiple domains, curate class-specific information, and employ large language models (LLMs) to construct an expert-tuning set, resulting in a 70k expert-tuning dataset called AgroInstruct. Subsequently, we expert-tuned and created AgroGPT, an efficient LMM that can hold complex agriculture-related conversations and provide useful insights. We also develop AgroEvals for evaluation and compare {AgroGPT's} performance with large open and closed-source models. {AgroGPT} excels at identifying fine-grained agricultural concepts, can act as an agriculture expert, and provides helpful information for multimodal agriculture questions. The code, datasets, and models are available at https://github.com/awaisrauf/agroGPT.