# MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents

arXiv: https://arxiv.org/abs/2508.11133 · HTML: https://mlsanity.com/paper/2508.11133/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning.
- Builds on 15 of its 71 references that are in the map.
- First posted on arXiv 15 Aug 2025.

## Placed in

- LLM reasoning / [Knowledge- and tool-augmented reasoning](https://mlsanity.com/lr/knowledge-and-tool-augmented-reasoning/)

## Builds on (most cited first, 15 of 15)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Humanity's Last Exam](https://mlsanity.com/paper/2501.14249/) (2025)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild](https://mlsanity.com/paper/2406.04770/) (2024)
- [Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.12941/) (2024)
- [AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?](https://mlsanity.com/paper/2407.15711/) (2024)
- [WideSearch: Benchmarking Agentic Broad Info-Seeking](https://mlsanity.com/paper/2508.07999/) (2025)
- [The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Input](https://mlsanity.com/paper/2501.03200/) (2025)
- [Towards Ecologically Valid Research on Language User Interfaces](https://mlsanity.com/paper/2007.14435/) (2020)
- [FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models](https://mlsanity.com/paper/2402.14116/) (2024)

## Built on by (most cited first, 21 of 21)

- [DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents](https://mlsanity.com/paper/2601.20975/) (2026)
- [ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics](https://mlsanity.com/paper/2509.00496/) (2025)
- [Oolong: Evaluating Long Context Reasoning and Aggregation Capabilities](https://mlsanity.com/paper/2511.02817/) (2025)
- [OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queries](https://mlsanity.com/paper/2605.06235/) (2026)
- [DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completeness](https://mlsanity.com/paper/2511.01323/) (2025)
- [When Is Enough Not Enough? Illusory Completion in Search Agents](https://mlsanity.com/paper/2602.07549/) (2026)
- [Structured RAG for Answering Aggregative Questions](https://mlsanity.com/paper/2511.08505/) (2025)
- [TodoEvolve: Learning to Architect Agent Planning Systems](https://mlsanity.com/paper/2602.07839/) (2026)
- [SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.08329/) (2026)
- [Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Length](https://mlsanity.com/paper/2603.22608/) (2026)
- [ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budget](https://mlsanity.com/paper/2604.01195/) (2026)
- [PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generation](https://mlsanity.com/paper/2606.28344/) (2026)
- [SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articles](https://mlsanity.com/paper/2509.21028/) (2025)
- [Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"](https://mlsanity.com/paper/2602.04853/) (2026)
- [A Benchmark for Deep Information Synthesis](https://mlsanity.com/paper/2602.21143/) (2026)
- [PluriHopRAG: Exhaustive, Recall-Sensitive QA Through Corpus-Specific Document Structure Learning](https://mlsanity.com/paper/2510.14377/) (2025)
- [Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)](https://mlsanity.com/paper/2606.05901/) (2026)
- [Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphs](https://mlsanity.com/paper/2607.25959/) (2026)
- [Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Search](https://mlsanity.com/paper/2608.20317/) (2026)
- [A Tree-based RAG Framework for Evidence-Intensive QA via Adaptive Planning and Topology-Aware Evidence Gathering](https://mlsanity.com/paper/2609.04981/) (2026)
- [PANINI-DR: Controlled Deep Research through Generative Semantic Workspaces](https://mlsanity.com/paper/2610.15208/) (2026)

## Abstract (arXiv)

Automated agents, powered by Large language models (LLMs), are emerging as the go-to tool for querying information. However, evaluation benchmarks for LLM agents rarely feature natural questions that are both information-seeking and genuinely time-consuming for humans. To address this gap we introduce MoNaCo, a benchmark of 1,315 natural and time-consuming questions that require dozens, and at times hundreds, of intermediate steps to solve -- far more than any existing QA benchmark. To build MoNaCo, we developed a decomposed annotation pipeline to elicit and manually answer real-world time-consuming questions at scale. Frontier LLMs evaluated on MoNaCo achieve at most 61.2% F1, hampered by low recall and hallucinations. Our results underscore the limitations of LLM-powered agents in handling the complexity and sheer breadth of real-world information-seeking tasks -- with MoNaCo providing an effective resource for tracking such progress. The MoNaCo benchmark, codebase, prompts and models predictions are all publicly available at: https://tomerwolgithub.github.io/monaco