mlsanityData to 7 Oct 2026

VLM4Bio: A Benchmark Dataset to Evaluate Pretrained Vision-Language Models for Trait Discovery from Biological Images

arXiv 2408.16176 ↗First posted 28 Aug 2024cs.CVcited by 10

On the map: Multimodal LLMs / vision-language models → Agricultural vision-language models

Abstract (arXiv)

Images are increasingly becoming the currency for documenting biodiversity on the planet, providing novel opportunities for accelerating scientific discoveries in the field of organismal biology, especially with the advent of large vision-language models (VLMs). We ask if pre-trained VLMs can aid scientists in answering a range of biologically relevant questions without any additional fine-tuning. In this paper, we evaluate the effectiveness of 12 state-of-the-art (SOTA) VLMs in the field of organismal biology using a novel dataset, VLM4Bio, consisting of 469K question-answer pairs involving 30K images from three groups of organisms: fishes, birds, and butterflies, covering five biologically relevant tasks. We also explore the effects of applying prompting techniques and tests for reasoning hallucination on the performance of VLMs, shedding new light on the capabilities of current SOTA VLMs in answering biologically relevant questions using images. The code and datasets for running all the analyses reported in this paper can be found at https://github.com/sammarfy/VLM4Bio.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Placed in Agricultural vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 82 references that are in the map. First posted on arXiv 28 Aug 2024.

Builds on

23 of 82 references.

References · 23

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  6. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  7. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  8. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  11. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
11 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  2. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  3. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 810
  4. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  5. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  6. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 677
  7. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  8. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  9. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  10. 2023Med-HALT: Medical Domain Hallucination Test for Large Language Modelscited by 156
  11. 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 2.

10 papers.

  1. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  2. 2025AgMMU: A Comprehensive Agricultural Multimodal Understanding Benchmarkcited by 15
  3. 2025The LLM Bottleneck: Why Open-Source Vision LLMs Struggle with Hierarchical Visual Recognitioncited by 7
  4. 2025Global and Local Entailment Learning for Natural World Imagerycited by 3
  5. 2025GRAID: Enhancing Spatial Reasoning of VLMs Through High-Fidelity Data Generationcited by 3
  6. 2026MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Modelscited by 3
  7. 2025AquaMonitor: A multimodal multi-view image sequence dataset for real-life aquatic invertebrate biodiversity monitoringcited by 0
  8. 2025AI in Agriculture: A Survey of Deep Learning Techniques for Crops, Fisheries and Livestockcited by 0
  9. 2025Visual Species Recognition with Large Multimodal Models as Post-Hoc Correctorscited by 0
  10. 2026A continental-scale dataset of ground beetles with high-resolution images and validated morphological trait measurementscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MarineEval: Assessing the Marine Intelligence of Vision-Language Models
  2. 2025PlantExpertVQA: A Visual Question Answering Dataset for Benchmarking Vision-Language Models in Plant Science
  3. 2025HoneyBee: Data Recipes for Vision-Language Reasoners
  4. 2026Benchmarking Vision-Language Models for Microscopic Plant Image Understanding
  5. 2025FishNet++: Analyzing the capabilities of Multimodal Large Language Models in marine biology
  6. 2026Can Edge-Deployable Vision-Language Models Identify Species?
  7. 2026LeafNet: A Large-Scale Dataset and Comprehensive Benchmark for Foundational Vision-Language Understanding of Plant Diseases
  8. 2026Thinking Like a Botanist: Challenging Multimodal Language Models with Intent-Driven Chain-of-Inquiry