mlsanityData to 7 Oct 2026

INQUIRE: A Natural World Text-to-Image Retrieval Benchmark

arXiv 2411.02537 ↗First posted 4 Nov 2024cs.CV cs.AI cs.CLcited by 33

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

We introduce INQUIRE, a text-to-image retrieval benchmark designed to challenge multimodal vision-language models on expert-level queries. INQUIRE includes iNaturalist 2024 (iNat24), a new dataset of five million natural world images, along with 250 expert-level retrieval queries. These queries are paired with all relevant images comprehensively labeled within iNat24, comprising 33,000 total matches. Queries span categories such as species identification, context, behavior, and appearance, emphasizing tasks that require nuanced image understanding and domain expertise. Our benchmark evaluates two core retrieval tasks: (1) INQUIRE-Fullrank, a full dataset ranking task, and (2) INQUIRE-Rerank, a reranking task for refining top-100 retrievals. Detailed evaluation of a range of recent multimodal models demonstrates that INQUIRE poses a significant challenge, with the best models failing to achieve an mAP@50 above 50%. In addition, we show that reranking with more powerful multimodal models can enhance retrieval performance, yet there remains a significant margin for improvement. By focusing on scientifically-motivated ecological challenges, INQUIRE aims to bridge the gap between AI capabilities and the needs of real-world scientific inquiry, encouraging the development of retrieval systems that can assist with accelerating ecological and biodiversity research. Our dataset and code are available at https://inquire-benchmark.github.io

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 79 references that are in the map. First posted on arXiv 4 Nov 2024.

Builds on

54 of 79 references.

References · 54

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  9. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  2. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2015VQA: Visual Question Answeringcited by 2,489
  6. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  9. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,163
  10. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,160
  11. 2023GAIA: a benchmark for General AI Assistantscited by 1,084
  12. 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
  13. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  14. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  15. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
  16. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  17. 2024PaliGemma: A versatile 3B VLM for transfercited by 679
  18. 2023DataComp: In search of the next generation of multimodal datasetscited by 635
  19. 2019Passage Re-ranking with BERTcited by 629
  20. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  21. 2020Google Landmarks Dataset v2 -- A Large-Scale Benchmark for Instance-Level Recognition and Retrievalcited by 305
  22. 2020Overview of the TREC 2019 deep learning trackcited by 282
  23. 2023Data Filtering Networkscited by 247
  24. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 238
  25. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  26. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  27. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  28. 2021Overview of the TREC 2020 deep learning trackcited by 193
  29. 2024An Introduction to Vision-Language Modelingcited by 175
  30. 2023LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmarkcited by 169
  31. 2021Benchmarking Representation Learning for Natural World Image Collectionscited by 167
  32. 2023BioCLIP: A Vision Foundation Model for the Tree of Lifecited by 159
  33. 2021Fine-Grained Image Analysis with Deep Learning: A Surveycited by 135
  34. 2023CLIPN for Zero-Shot OOD Detection: Teaching CLIP to Say Nocited by 130
  35. 2023Encyclopedic VQA: Visual questions about detailed properties of fine-grained categoriescited by 105
  36. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  37. 2023COLA: A Benchmark for Compositional Text-to-image Retrievalcited by 53
  38. 2020ProtoQA: A Question Answering Dataset for Prototypical Common-Sense Reasoningcited by 45

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 18; 2026: 15.

33 papers.

Sort:

By area:

  1. 2025BioCLIP 2: Emergent Properties from Scaling Hierarchical Contrastive Learningcited by 48
  2. 2025ImageRAG: Dynamic Image Retrieval for Reference-Guided Image Generationcited by 20
  3. 2025Visual-RAG: Benchmarking Text-to-Image Retrieval Augmented Generation for Visual Knowledge Intensive Queriescited by 16
  4. 2025ELIP: Enhanced Visual-Language Foundation Models for Image Retrievalcited by 9
  5. 2026HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agentscited by 8
  6. 2025WeatherPrompt: Multi-modality Representation Learning for All-Weather Drone Visual Geo-Localizationcited by 7
  7. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  8. 2025Same or Not? Enhancing Visual Perception in Vision-Language Modelscited by 5
  9. 2025VisRet: Visualization Improves Knowledge-Intensive Text-to-Image Retrievalcited by 3
  10. 2025Global and Local Entailment Learning for Natural World Imagerycited by 3
  11. 2025BioCAP: Exploiting Synthetic Captions Beyond Labels in Biological Foundation Modelscited by 3
  12. 2025MUVR: A Multi-Modal Untrimmed Video Retrieval Benchmark with Multi-Level Visual Correspondencecited by 3
21 more
  1. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledgecited by 3
  2. 2025Open-Insect: Benchmarking Open-Set Recognition of Novel Species in Biodiversity Monitoringcited by 2
  3. 2025QuARI: Query Adaptive Retrieval Improvementcited by 2
  4. 2025Robustness in Both Domains: CLIP Needs a Robust Text Encodercited by 2
  5. 2026Autonomous Scientific Discovery via Iterative Meta-Reflectioncited by 2
  6. 2025AnimalClue: Recognizing Animals by their Tracescited by 1
  7. 2025ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecologycited by 1
  8. 2025Semantic search for 100M+ galaxy images using AI-generated captionscited by 1
  9. 2026Few Shots Text to Image Retrieval: New Benchmarking Dataset and Optimization Methodscited by 1
  10. 2026Energy-Efficient Plant Monitoring via Knowledge Distillationcited by 1
  11. 2025INQUIRE-Search: Interactive Discovery in Large-Scale Biodiversity Databasescited by 0
  12. 2026Compact Hypercube Embeddings for Fast Text-based Wildlife Observation Retrievalcited by 0
  13. 2026Audio-to-Image Bird Species Retrieval without Audio-Image Pairs via Text Distillationcited by 0
  14. 2026Approximate Nearest Neighbor Search for Modern AI: A Projection-Augmented Graph Approachcited by 0
  15. 2026eLasmobranc Dataset: An Image Dataset for Elasmobranch Species Recognition and Biodiversity Monitoringcited by 0
  16. 2026BioVITA: Biological Dataset, Model, and Benchmark for Visual-Textual-Acoustic Alignmentcited by 0
  17. 2026RealBirdID: Benchmarking Bird Species Identification in the Era of MLLMscited by 0
  18. 2026PRISMR: Overcoming Parse Collapse in Multimodal Listwise Ranking via Parameterized Representation Internalizationcited by 0
  19. 2026WildProp: Visual Estimation of Wildlife Body Proportions at Scalecited by 0
  20. 2026Prompting-MammAlps: Fine-Grained Text-to-Video Retrieval for Camera-Trap Datacited by 0
  21. 2026Rethinking Text-Based Image Retrieval in Specific Domaincited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  2. 2024Benchmark Granularity and Model Robustness for Image-Text Retrieval
  3. 2025Vision-Free Retrieval: Rethinking Multimodal Search with Textual Scene Descriptions
  4. 2025Chain-of-Thought Re-ranking for Image Retrieval Tasks
  5. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  6. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
  7. 2026EviRank: Structured Relevance Evidence for Multimodal Image Re-ranking
  8. 2025ImageScope: Unifying Language-Guided Image Retrieval via Large Multimodal Model Collective Reasoning