mlsanityData to 14 Oct 2026

MoNaCo: More Natural and Complex Questions for Reasoning Across Dozens of Documents

arXiv 2508.11133 ↗First posted 15 Aug 2025cs.CL cs.AI cs.DBcited by 21

On the map: LLM reasoning → Knowledge- and tool-augmented reasoning

Abstract (arXiv)

Automated agents, powered by Large language models (LLMs), are emerging as the go-to tool for querying information. However, evaluation benchmarks for LLM agents rarely feature natural questions that are both information-seeking and genuinely time-consuming for humans. To address this gap we introduce MoNaCo, a benchmark of 1,315 natural and time-consuming questions that require dozens, and at times hundreds, of intermediate steps to solve -- far more than any existing QA benchmark. To build MoNaCo, we developed a decomposed annotation pipeline to elicit and manually answer real-world time-consuming questions at scale. Frontier LLMs evaluated on MoNaCo achieve at most 61.2% F1, hampered by low recall and hallucinations. Our results underscore the limitations of LLM-powered agents in handling the complexity and sheer breadth of real-world information-seeking tasks -- with MoNaCo providing an effective resource for tracking such progress. The MoNaCo benchmark, codebase, prompts and models predictions are all publicly available at: https://tomerwolgithub.github.io/monaco

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Knowledge- and tool-augmented reasoning, one of 17 directions in LLM reasoning. Builds on 15 of its 71 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

15 of 71 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation19 Sep

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2024OpenAI o1 System Cardcited by 1,244
  6. 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 647
  7. 2025Humanity's Last Examcited by 491
  8. 2024Measuring short-form factuality in large language modelscited by 373
  9. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  10. 2024AssistantBench: Can Web Agents Solve Realistic and Time-Consuming Tasks?cited by 158
  11. 2025WideSearch: Benchmarking Agentic Broad Info-Seekingcited by 69
  12. 2025The FACTS Grounding Leaderboard: Benchmarking LLMs' Ability to Ground Responses to Long-Form Inputcited by 52
2 more
  1. 2020Towards Ecologically Valid Research on Language User Interfacescited by 42
  2. 2024FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Modelscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 15.

21 papers.

Sort:

By area:

  1. 2026DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agentscited by 50
  2. 2025ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubricscited by 49
  3. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  4. 2026OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queriescited by 8
  5. 2025DEEPAMBIGQA: Ambiguous Multi-hop Questions for Benchmarking LLM Answer Completenesscited by 6
  6. 2026When Is Enough Not Enough? Illusory Completion in Search Agentscited by 5
  7. 2025Structured RAG for Answering Aggregative Questionscited by 3
  8. 2026TodoEvolve: Learning to Architect Agent Planning Systemscited by 2
  9. 2026SPD-RAG: Sub-Agent Per Document Retrieval-Augmented Generationcited by 2
  10. 2026Understanding LLM Performance Degradation in Multi-Instance Processing: The Roles of Instance Count and Context Lengthcited by 2
  11. 2026ORBIT: Scalable and Verifiable Data Generation for Search Agents on a Tight Budgetcited by 2
  12. 2026PIXELRAG: Web Screenshots Beat Text for Retrieval-Augmented Generationcited by 2
9 more
  1. 2025SciTrek: Evaluating and Improving Long-Context Numerical Reasoning over Scientific Articlescited by 1
  2. 2026Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know"cited by 1
  3. 2026A Benchmark for Deep Information Synthesiscited by 1
  4. 2025PluriHopRAG: Exhaustive, Recall-Sensitive QA Through Corpus-Specific Document Structure Learningcited by 0
  5. 2026Reducing Hallucinations in Complex Question Answering using Simple Graph-based Retrieval-Augmented Generation (long version)cited by 0
  6. 2026Detecting Knowledge Inconsistencies Across Text, Tables, and Knowledge Graphscited by 0
  7. 2026Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Searchcited by 0
  8. 2026A Tree-based RAG Framework for Evidence-Intensive QA via Adaptive Planning and Topology-Aware Evidence Gatheringcited by 0
  9. 2026PANINI-DR: Controlled Deep Research through Generative Semantic Workspacescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey of Large Language Model Agents for Question Answering
  2. 2026LakeQA: An Exploratory QA Benchmark over a Million-Scale Data Lake
  3. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
  4. 2025MSCoRe: A Benchmark for Multi-Stage Collaborative Reasoning in LLM Agents
  5. 2025Agentic LLMs for Question Answering over Tabular Data
  6. 2025JudgeAgent: Beyond Static Benchmarks for Knowledge-Driven and Dynamic LLM Evaluation
  7. 2023On Evaluating the Integration of Reasoning and Action in LLM Agents with Database Question Answering
  8. 2024MINTQA: A Multi-Hop Question Answering Benchmark for Evaluating LLMs on New and Tail Knowledge