mlsanityData to 7 Oct 2026

HIVE: Query, Hypothesize, Verify An LLM Framework for Multimodal Reasoning-Intensive Retrieval

arXiv 2604.07220 ↗First posted 8 Apr 2026cs.IRcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Multimodal retrieval models fail on reasoning-intensive queries where images (diagrams, charts, screenshots) must be deeply integrated with text to identify relevant documents -- the best multimodal model achieves only 27.6 nDCG@10 on MM-BRIGHT, underperforming even strong text-only retrievers (32.2). We introduce HIVE (Hypothesis-driven Iterative Visual Evidence Retrieval), a plug-and-play framework that injects explicit visual-text reasoning into a retriever via LLMs. HIVE operates in four stages: (1) initial retrieval over the corpus, (2) LLM-based compensatory query synthesis that explicitly articulates visual and logical gaps observed in top-k candidates, (3) secondary retrieval with the refined query, and (4) LLM verification and reranking over the union of candidates. Evaluated on the multimodal-to-text track of MM-BRIGHT (2,803 real-world queries across 29 technical domains), HIVE achieves a new state-of-the-art aggregated nDCG@10 of 41.7 -- a +9.5 point gain over the best text-only model (DiVeR: 32.2) and +14.1 over the best multimodal model (Nomic-Vision: 27.6), where our reasoning-enhanced base retriever contributes 33.2 and the HIVE framework adds a further +8.5 points -- with particularly strong results in visually demanding domains (Gaming: 68.2, Chemistry: 42.5, Sustainability: 49.4). Compatible with both standard and reasoning-enhanced retrievers, HIVE demonstrates that LLM-mediated visual hypothesis generation and verification can substantially close the multimodal reasoning gap in retrieval. https://github.com/mm-bright/multimodal-reasoning-retrieval

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 24 of its 43 references that are in the map. First posted on arXiv 8 Apr 2026.

Builds on

24 of 43 references.

References · 24

  1. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  2. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  3. 2024Multilingual E5 Text Embeddings: A Technical Reportcited by 357
  4. 2024VisRAG: Vision-based Retrieval-augmented Generation on Multi-modality Documentscited by 237
  5. 2024VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Taskscited by 222
  6. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
  7. 2024ColPali: Efficient Document Retrieval with Vision Language Modelscited by 190
  8. 2024GME: Improving Universal Multimodal Retrieval by Multimodal LLMscited by 166
  9. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 153
  10. 2024MegaPairs: Massive Data Synthesis For Universal Multimodal Retrievalcited by 84
  11. 2025ReasonIR: Training Retrievers for Reasoning Taskscited by 76
  12. 2025Rank1: Test-Time Compute for Reranking in Information Retrievalcited by 68
12 more
  1. 2025Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learningcited by 58
  2. 2024jina-clip-v2: Multilingual Multimodal Embeddings for Text and Imagescited by 36
  3. 2025DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrievalcited by 35
  4. 2025RaDeR: Reasoning-aware Dense Retrieval Modelscited by 27
  5. 2024Nomic Embed Vision: Expanding the Latent Spacecited by 19
  6. 2025Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generationcited by 13
  7. 2025ThinkQE: Query Expansion via an Evolving Thinking Processcited by 11
  8. 2025DeAR: Dual-Stage Document Reranking with Reasoning Agents via LLM Distillationcited by 11
  9. 2026MM-BRIGHT: A Multi-Task Multimodal Benchmark for Reasoning-Intensive Retrievalcited by 11
  10. 2025MR2-Bench: Going Beyond Matching to Reasoning in Multimodal Retrievalcited by 7
  11. 2026TEMPO: A Realistic Multi-Domain Benchmark for Temporal Reasoning-Intensive Retrievalcited by 6
  12. 2025SustainableQA: A Comprehensive Question Answering Dataset for Corporate Sustainability and EU Taxonomy Reportingcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MARVEL: Multimodal Adaptive Reasoning-intensiVe Expand-rerank and retrievaL
  2. 2026ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge
  3. 2026CrossModalQA: A Cross-modal and Multi-hop Benchmark for Multimodal Retrieval-augmented Generation
  4. 2026Reasoning-Augmented Representations for Multimodal Retrieval
  5. 2025Reason to Contrast: A Cascaded Multimodal Retrieval Framework
  6. 2025UNIDOC-BENCH: A Unified Benchmark for Document-Centric Multimodal RAG
  7. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
  8. 2025MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval