# MRMR: A Realistic and Expert-Level Multidisciplinary Benchmark for Reasoning-Intensive Multimodal Retrieval

arXiv: https://arxiv.org/abs/2510.09510 · HTML: https://mlsanity.com/paper/2510.09510/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 30 of its 72 references that are in the map.
- First posted on arXiv 10 Oct 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)

## Builds on (most cited first, 30 of 30)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [Qwen-Image Technical Report](https://mlsanity.com/paper/2508.02324/) (2025)
- [EVA-CLIP: Improved Training Techniques for CLIP at Scale](https://mlsanity.com/paper/2303.15389/) (2023)
- [Kimi K2: Open Agentic Intelligence](https://mlsanity.com/paper/2507.20534/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models](https://mlsanity.com/paper/2405.17428/) (2024)
- [BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent](https://mlsanity.com/paper/2508.06600/) (2025)
- [MMSearch-R1: Incentivizing LMMs to Search](https://mlsanity.com/paper/2506.20670/) (2025)
- [MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructions](https://mlsanity.com/paper/2403.19651/) (2024)
- [Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities](https://mlsanity.com/paper/2302.11154/) (2023)
- [MMSearch: Benchmarking the Potential of Large Models as Multi-modal Search Engines](https://mlsanity.com/paper/2409.12959/) (2024)
- [MonkeyOCR: Document Parsing with a Structure-Recognition-Relation Triplet Paradigm](https://mlsanity.com/paper/2506.05218/) (2025)
- [MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot](https://mlsanity.com/paper/2502.04413/) (2025)
- [ReasonIR: Training Retrievers for Reasoning Tasks](https://mlsanity.com/paper/2504.20595/) (2025)
- [Rank1: Test-Time Compute for Reranking in Information Retrieval](https://mlsanity.com/paper/2502.18418/) (2025)
- [WebResearcher: Unleashing unbounded reasoning capability in Long-Horizon Agents](https://mlsanity.com/paper/2509.13309/) (2025)
- [Seedream 2.0: A Native Chinese-English Bilingual Image Generation Foundation Model](https://mlsanity.com/paper/2503.07703/) (2025)
- [ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval](https://mlsanity.com/paper/2505.17166/) (2025)
- [MMDocIR: Benchmarking Multimodal Retrieval for Long Documents](https://mlsanity.com/paper/2501.08828/) (2025)
- [jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images](https://mlsanity.com/paper/2412.08802/) (2024)
- [DIVER: A Multi-Stage Approach for Reasoning-intensive Information Retrieval](https://mlsanity.com/paper/2508.07995/) (2025)
- [REAL-MM-RAG: A Real-World Multi-Modal Retrieval Benchmark](https://mlsanity.com/paper/2502.12342/) (2025)
- [EDIS: Entity-Driven Image Search over Multimodal Web Content](https://mlsanity.com/paper/2305.13631/) (2023)
- [RaDeR: Reasoning-aware Dense Retrieval Models](https://mlsanity.com/paper/2505.18405/) (2025)
- [Diffusion vs. Autoregressive Language Models: A Text Embedding Perspective](https://mlsanity.com/paper/2505.15045/) (2025)
- [MIEB: Massive Image Embedding Benchmark](https://mlsanity.com/paper/2504.10471/) (2025)
- [MMRAG: Multi-Mode Retrieval-Augmented Generation with Large Language Models for Biomedical In-Context Learning](https://mlsanity.com/paper/2502.15954/) (2025)

## Built on by (most cited first, 12 of 12)

- [Beyond Chain-of-Thought: Rewrite as a Universal Interface for Generative Multimodal Embeddings](https://mlsanity.com/paper/2604.22280/) (2026)
- [LimRank: Less is More for Reasoning-Intensive Information Reranking](https://mlsanity.com/paper/2510.23544/) (2025)
- [ARK: A Dual-Axis Multimodal Retrieval Benchmark along Reasoning and Knowledge](https://mlsanity.com/paper/2602.09839/) (2026)
- [Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems](https://mlsanity.com/paper/2605.04018/) (2026)
- [ReasonAudio: A Benchmark for Evaluating Reasoning Beyond Matching in Text-Audio Retrieval](https://mlsanity.com/paper/2605.03361/) (2026)
- [XDomainBench: Diagnosing Reasoning Collapse in High-Dimensional Scientific Knowledge Composition](https://mlsanity.com/paper/2605.14754/) (2026)
- [From "Weak" Signals to Strong Models: Preference Delta Aggregation with LoRA Merging](https://mlsanity.com/paper/2606.00357/) (2026)
- [ReLoop-UME: Recurrent Depth with Learnable Retrieval Registers for Universal Multimodal Embedding](https://mlsanity.com/paper/2607.28751/) (2026)
- [CORE: Improving Compositional Reasoning in MLLM Embedding via Reranker Distillation](https://mlsanity.com/paper/2609.04083/) (2026)
- [MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models](https://mlsanity.com/paper/2609.08663/) (2026)
- [Reason What Matters: Retrieval-Grounded Reasoning for Universal Multimodal Embeddings](https://mlsanity.com/paper/2609.15296/) (2026)
- [VaME: Exploring Variational Latent Reasoning for Multimodal Embeddings](https://mlsanity.com/paper/2609.33402/) (2026)

## Abstract (arXiv)

We introduce MRMR, the first expert-level multidisciplinary multimodal retrieval benchmark requiring intensive reasoning. MRMR contains 1,502 queries spanning 23 domains, with positive documents carefully verified by human experts. Compared to prior benchmarks, MRMR introduces three key advancements. First, it challenges retrieval systems across diverse areas of expertise, enabling fine-grained model comparison across domains. Second, queries are reasoning-intensive, with images requiring deeper interpretation such as diagnosing microscopic slides. We further introduce Contradiction Retrieval, a novel task requiring models to identify conflicting concepts. Finally, queries and documents are constructed as image-text interleaved sequences. Unlike earlier benchmarks restricted to single images or unimodal documents, MRMR offers a realistic setting with multi-image queries and mixed-modality corpus documents. We conduct an extensive evaluation of 4 categories of multimodal retrieval systems and 14 frontier models on MRMR. The text embedding model Qwen3-Embedding with LLM-generated image captions achieves the highest performance, highlighting substantial room for improving multimodal retrieval models. Although latest multimodal models such as Ops-MM-Embedding perform competitively on expert-domain queries, they fall short on reasoning-intensive tasks. We believe that MRMR paves the way for advancing multimodal retrieval in more realistic and challenging scenarios.