mlsanityData to 14 Oct 2026

PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluation

arXiv 2502.20377 ↗First posted 27 Feb 2025cs.LG cs.AI cs.CLcited by 14

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

High-quality benchmarks are essential for evaluating reasoning and retrieval capabilities of large language models (LLMs). However, curating datasets for this purpose is not a permanent solution as they are prone to data leakage and inflated performance results. To address these challenges, we propose PhantomWiki: a pipeline to generate unique, factually consistent document corpora with diverse question-answer pairs. Unlike prior work, PhantomWiki is neither a fixed dataset, nor is it based on any existing data. Instead, a new PhantomWiki instance is generated on demand for each evaluation. We vary the question difficulty and corpus size to disentangle reasoning and retrieval capabilities respectively, and find that PhantomWiki datasets are surprisingly challenging for frontier LLMs. Thus, we contribute a scalable and data leakage-resistant framework for disentangled evaluation of reasoning, retrieval, and tool-use abilities. Our code is available at https://github.com/kilian-group/phantom-wiki.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 69 of its 114 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

69 of 114 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023MuSR: Testing the Limits of Chain-of-thought with Multistep Soft Reasoning24 Oct
  2. 2023Gorilla: Large Language Model Connected with Massive APIs24 May
  3. 2022Measuring and Narrowing the Compositionality Gap in Language Models7 Oct
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Models6 Oct
  5. 2022Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought3 Oct
  6. 2022FOLIO: Natural Language Reasoning with First-Order Logic2 Sep

Other references · 63

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2024GPT-4o System Cardcited by 4,588
  10. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
  11. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  12. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
38 more
  1. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  8. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  9. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  10. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  11. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  12. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  13. 2020ALFWorld: Aligning Text and Embodied Environments for Interactive Learningcited by 1,205
  14. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  15. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  16. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  17. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  18. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  19. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  20. 2024GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Modelscited by 550
  21. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  22. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  23. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  24. 2020ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Languagecited by 324
  25. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  26. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 314
  27. 2023ToolQA: A Dataset for LLM Question Answering with External Toolscited by 274
  28. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  29. 2020HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Datacited by 264
  30. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  31. 2022RealTime QA: What's the Answer Right Now?cited by 244
  32. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 242
  33. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  34. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  35. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  36. 2017Constructing Datasets for Multi-hop Reading Comprehension Across Documentscited by 190
  37. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Researchcited by 166
  38. 2019CLUTRR: A Diagnostic Benchmark for Inductive Reasoning from Textcited by 162

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 12.

14 papers.

Sort:

By area:

  1. 2025NarraBench: A Comprehensive Framework for Narrative Benchmarkingcited by 13
  2. 2026Learning from Synthetic Data Improves Multi-hop Reasoningcited by 6
  3. 2025CausalARC: Abstract Reasoning with Causal World Modelscited by 4
  4. 2026MixSD: Mixed Contextual Self-Distillation for Knowledge Injectioncited by 3
  5. 2026Deep Reasoning in General Purpose Agents via Structured Meta-Cognitioncited by 2
  6. 2026Bridging Structure and Language: Graph-Based Visual Reasoning for Autonomous Road Understandingcited by 2
  7. 2026Copy Less, Ground More: Overcoming Repetitive Copying in Long-Context Reasoning via Evidence-Aware Reinforcement Learningcited by 1
  8. 2026CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Basescited by 1
  9. 2026Training a Knowledge Base: Supervised Structure Learning for Agent-Curated Document Storescited by 0
  10. 2026Evidence Blindness in Direct Corpus Interaction: Persistent Navigation with AtlasNavcited by 0
  11. 2026SHELF: A Synthetic Harness for Multi-Task Bibliographic Benchmarkingcited by 0
  12. 2026A Four-Stage Decomposition of Word-Problem Solving and Mechanistic Fragility in LLM Math Reasoningcited by 0
2 more
  1. 2026Learning to Route in Visual Space via Multi-Step Embedding Retrievalcited by 0
  2. 2026PhantomEnvironments: Training LLM Agents in Fictional Worldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CorpusQA: A 10 Million Token Benchmark for Corpus-Level Analysis and Reasoning
  2. 2025CoLoTa: A Dataset for Entity-based Commonsense Reasoning over Long-Tail Knowledge
  3. 2026CommonWhy: A Dataset for Evaluating Entity-Based Causal Commonsense Reasoning in Large Language Models
  4. 2024FanOutQA: A Multi-Hop, Multi-Document Question Answering Benchmark for Large Language Models
  5. 2022WikiWhy: Answering and Explaining Cause-and-Effect Questions
  6. 2023DIVKNOWQA: Assessing the Reasoning Ability of LLMs via Open-Domain Question Answering over Knowledge Base and Text
  7. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
  8. 2026Breaking the Evaluation Paradox: Evaluating High-Entropy Search with Computationally Irreducible Constraints