# MERRIN: A Benchmark for Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments

arXiv: https://arxiv.org/abs/2604.13418 · HTML: https://mlsanity.com/paper/2604.13418/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 25 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 20 of 20)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Knowledge Conflicts for LLMs: A Survey](https://mlsanity.com/paper/2403.08319/) (2024)
- [Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?](https://mlsanity.com/paper/2302.11713/) (2023)
- [WebWatcher: Breaking New Frontier of Vision-Language Deep Research Agent](https://mlsanity.com/paper/2508.05748/) (2025)
- [SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models](https://mlsanity.com/paper/2506.01062/) (2025)
- [Retrieval-Augmented Generation with Conflicting Evidence](https://mlsanity.com/paper/2504.13079/) (2025)
- [MM-BrowseComp: A Comprehensive Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.13186/) (2025)
- [Attacking Open-domain Question Answering by Injecting Misinformation](https://mlsanity.com/paper/2110.07803/) (2021)
- [MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents](https://mlsanity.com/paper/2508.21475/) (2025)
- [ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models](https://mlsanity.com/paper/2505.13444/) (2025)
- [Multimodal Inconsistency Reasoning (MMIR): A New Benchmark for Multimodal Reasoning Models](https://mlsanity.com/paper/2502.16033/) (2025)
- [Benchmarking Multimodal Knowledge Conflict for Large Multimodal Models](https://mlsanity.com/paper/2505.19509/) (2025)
- [How Well Do Large Language Models Truly Ground?](https://mlsanity.com/paper/2311.09069/) (2023)
- [Mitigating Modal Imbalance in Multimodal Reasoning](https://mlsanity.com/paper/2510.02608/) (2025)
- [BrowseComp-V3: A Visual, Vertical, and Verifiable Benchmark for Multimodal Browsing Agents](https://mlsanity.com/paper/2602.12876/) (2026)
- [Detecting Corpus-Level Knowledge Inconsistencies in Wikipedia with Large Language Models](https://mlsanity.com/paper/2509.23233/) (2025)
- [CORG: Generating Answers from Complex, Interrelated Contexts](https://mlsanity.com/paper/2505.00023/) (2025)
- [CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolution](https://mlsanity.com/paper/2511.21717/) (2025)

## Built on by (most cited first, 3 of 3)

- [FIKA-Bench: From Fine-grained Recognition to Fine-Grained Knowledge Acquisition](https://mlsanity.com/paper/2605.13193/) (2026)
- [MMShopBench: A Real-Log Benchmark for Multimodal, Multi-Turn Shopping Agents](https://mlsanity.com/paper/2607.29002/) (2026)
- [HyperBrowseComp: A Multilingual and Multimodal Stress Test for Web-Browsing Agents](https://mlsanity.com/paper/2610.03574/) (2026)

## Abstract (arXiv)

Motivated by the underspecified, multi-hop nature of search queries and the multimodal, heterogeneous, and often conflicting nature of real-world web results, we introduce MERRIN (Multimodal Evidence Retrieval and Reasoning in Noisy Web Environments), a human-annotated benchmark for evaluating search-augmented agents. MERRIN measures AI agents' ability to identify relevant modalities, retrieve multimodal evidence, and perform multi-hop reasoning over noisy web sources. It differs from prior work in three important aspects: (1) using natural language queries without explicit modality cues, (2) incorporating underexplored modalities such as video and audio, and (3) requiring the retrieval of complex, often noisy or conflicting multimodal evidence during web search. We evaluate diverse search agents powered by ten models, including strong closed-source models (e.g., GPT-5.4-mini, Gemini 3/3.1 Flash/Pro) and open-weight models (Qwen3-4B/30B/235B), across three search settings (no search, native search, and agentic search). Our results show that MERRIN is highly challenging: the average accuracy across all agents is 22.3%, with the best-performing agent reaching only 40.1%. We further observe that while stronger agents like Gemini Deep Research achieve higher performance, gains are modest due to over-exploration; they take more steps and use more tools, but are often distracted by conflicting or partially relevant web content, leading to incorrect answers. Compared to humans, these agents consume more resources yet achieve lower accuracy, largely due to inefficient source selection and an overreliance on text modalities. These findings highlight the need for search agents capable of robust search and reasoning across diverse modalities in noisy web environments, making MERRIN a valuable testbed for evaluating such capabilities.