# Benchmarking In-the-wild Multimodal Disease Recognition and A Versatile Baseline

arXiv: https://arxiv.org/abs/2408.03120 · HTML: https://mlsanity.com/paper/2408.03120/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 6 Aug 2024.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [CLIP-Adapter: Better Vision-Language Models with Feature Adapters](https://mlsanity.com/paper/2110.04544/) (2021)
- [Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigm](https://mlsanity.com/paper/2110.05208/) (2021)
- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [VirTex: Learning Visual Representations from Textual Annotations](https://mlsanity.com/paper/2006.06666/) (2020)
- [PointCLIP V2: Prompting CLIP and GPT for Powerful 3D Open-world Learning](https://mlsanity.com/paper/2211.11682/) (2022)
- [An open access repository of images on plant health to enable the development of mobile disease diagnostics](https://mlsanity.com/paper/1511.08060/) (2015)
- [CALIP: Zero-Shot Enhancement of CLIP with Parameter-free Attention](https://mlsanity.com/paper/2209.14169/) (2022)
- [A Good Prompt Is Worth Millions of Parameters: Low-resource Prompt-based Learning for Vision-Language Models](https://mlsanity.com/paper/2110.08484/) (2021)
- [CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIP](https://mlsanity.com/paper/2110.11316/) (2021)
- [PlantDoc: A Dataset for Visual Plant Disease Detection](https://mlsanity.com/paper/1911.10317/) (2019)

## Built on by (most cited first, 19 of 19)

- [AgroBench: Vision-Language Model Benchmark in Agriculture](https://mlsanity.com/paper/2507.20519/) (2025)
- [AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmark](https://mlsanity.com/paper/2504.10568/) (2025)
- [PlantSeg: A Large-Scale In-the-wild Dataset for Plant Disease Segmentation](https://mlsanity.com/paper/2409.04038/) (2024)
- [Cluster-Aware Prompt Ensemble Learning for Few-Shot Vision-Language Model Adaptation](https://mlsanity.com/paper/2510.09867/) (2025)
- [SVIP: Semantically Contextualized Visual Patches for Zero-Shot Learning](https://mlsanity.com/paper/2503.10252/) (2025)
- [CF-PRNet: Coarse-to-Fine Prototype Refining Network for Point Cloud Completion and Reconstruction](https://mlsanity.com/paper/2409.08443/) (2024)
- [Track Any Peppers: Weakly Supervised Sweet Pepper Tracking Using VLMs](https://mlsanity.com/paper/2411.06702/) (2024)
- [Plant Disease Detection through Multimodal Large Language Models and Convolutional Neural Networks](https://mlsanity.com/paper/2504.20419/) (2025)
- [Augment to Segment: Tackling Pixel-Level Imbalance in Wheat Disease and Pest Segmentation](https://mlsanity.com/paper/2509.09961/) (2025)
- [Distributed Zero-Shot Learning for Visual Recognition](https://mlsanity.com/paper/2511.08170/) (2025)
- [Learning Multi-Modal Prototypes for Cross-Domain Few-Shot Object Detection](https://mlsanity.com/paper/2602.18811/) (2026)
- [Mobile-Friendly Deep Learning for Plant Disease Detection: A Lightweight CNN Benchmark Across 101 Classes of 33 Crops](https://mlsanity.com/paper/2508.10817/) (2025)
- [MITS: A Large-Scale Multimodal Benchmark Dataset for Intelligent Traffic Surveillance](https://mlsanity.com/paper/2509.09730/) (2025)
- [Energy-Efficient Plant Monitoring via Knowledge Distillation](https://mlsanity.com/paper/2604.27178/) (2026)
- [StomataSeg: Semi-Supervised Instance Segmentation for Sorghum Stomatal Components](https://mlsanity.com/paper/2602.00703/) (2026)
- [CropVLM: A Domain-Adapted Vision-Language Model for Open-Set Crop Analysis](https://mlsanity.com/paper/2605.03259/) (2026)
- [Benchmarking Vision-Language Models for Microscopic Plant Image Understanding](https://mlsanity.com/paper/2606.22497/) (2026)
- [AppleScab-LT: A Longitudinal Real-Field Apple Scab Dataset for Temporal Disease Progression Analysis](https://mlsanity.com/paper/2608.14235/) (2026)
- [CropCop: An Auditable 120-Class Plant-Health Model from Benchmark Reconstruction to a Quantised Runtime Artifact](https://mlsanity.com/paper/2608.25539/) (2026)

## Abstract (arXiv)

Existing plant disease classification models have achieved remarkable performance in recognizing in-laboratory diseased images. However, their performance often significantly degrades in classifying in-the-wild images. Furthermore, we observed that in-the-wild plant images may exhibit similar appearances across various diseases (i.e., small inter-class discrepancy) while the same diseases may look quite different (i.e., large intra-class variance). Motivated by this observation, we propose an in-the-wild multimodal plant disease recognition dataset that contains the largest number of disease classes but also text-based descriptions for each disease. Particularly, the newly provided text descriptions are introduced to provide rich information in textual modality and facilitate in-the-wild disease classification with small inter-class discrepancy and large intra-class variance issues. Therefore, our proposed dataset can be regarded as an ideal testbed for evaluating disease recognition methods in the real world. In addition, we further present a strong yet versatile baseline that models text descriptions and visual data through multiple prototypes for a given class. By fusing the contributions of multimodal prototypes in classification, our baseline can effectively address the small inter-class discrepancy and large intra-class variance issues. Remarkably, our baseline model can not only classify diseases but also recognize diseases in few-shot or training-free scenarios. Extensive benchmarking results demonstrate that our proposed in-the-wild multimodal dataset sets many new challenges to the plant disease recognition task and there is a large space to improve for future works.