mlsanityData to 14 Oct 2026

PRELUDE: A Benchmark Designed to Require Global Comprehension and Reasoning over Long Contexts

arXiv 2508.09848 ↗First posted 13 Aug 2025cs.CL cs.AIcited by 7

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

We introduce PRELUDE, a benchmark for evaluating long-context understanding through the task of determining whether a character's prequel story is consistent with the canonical narrative of the original book. Our task poses a stronger demand for global comprehension and deep reasoning than existing benchmarks -- as the prequels are not part of the original story, assessing their plausibility typically requires searching and integrating information that is only indirectly related. Empirically, 88% of instances require evidence from multiple parts of the narrative. Experimental results highlight the challenge of our task: in-context learning, RAG and in-domain training with state-of-the-art LLMs, and commercial DeepResearch services, lag behind humans by >15%. A further human study reveals that models often produce correct answers with flawed reasoning, leading to an over 30% gap in reasoning accuracy compared to humans. These findings underscore the substantial room for improvement in long-context understanding and reasoning.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 27 of its 66 references that are in the map. First posted on arXiv 13 Aug 2025.

Builds on

27 of 66 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitasks19 Dec
  2. 2024NeedleBench: Evaluating LLM Retrieval and Reasoning Across Varying Information Densities16 Jul

Other references · 25

  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  2. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  3. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2025s1: Simple test-time scalingcited by 1,341
  6. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  7. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  8. 2020Fact or Fiction: Verifying Scientific Claimscited by 554
  9. 2025RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learningcited by 386
  10. 2023Language Modeling Is Compressioncited by 225
  11. 2025ARC-AGI-2: A New Challenge for Frontier AI Reasoning Systemscited by 190
  12. 2019On the Measure of Intelligencecited by 144
13 more
  1. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  2. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  3. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  4. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 69
  5. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokenscited by 32
  6. 2025LaRA: Benchmarking Retrieval-Augmented Generation and Long-Context LLMs -- No Silver Bullet for LC or RAG Routingcited by 29
  7. 2023Personality Understanding of Fictional Characters during Book Readingcited by 20
  8. 2024XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependenciescited by 14
  9. 2025Shifting Long-Context LLMs Research from Input to Outputcited by 14
  10. 2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
  11. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
  12. 2025The Essence of Contextual Understanding in Theory of Mind: A Study on Question Answering with Story Characterscited by 8
  13. 2025SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehensioncited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 5.

7 papers.

  1. 2025HGMEM: Hypergraph-based Working Memory to Improve Multi-step RAG for Long-Context Complex Relational Modelingcited by 4
  2. 2026Many-Shot CoT-ICL: Making In-Context Learning Truly Learncited by 3
  3. 2026A New Role for Relevance: Guiding Corpus Interaction in Agentic Searchcited by 3
  4. 2025DivLogicEval: A Framework for Benchmarking Logical Reasoning Evaluation in Large Language Modelscited by 2
  5. 2026Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answerscited by 0
  6. 2026Towards High-Level Semantic Intelligencecited by 0
  7. 2026InsightEmb: Learning Action-Intent Embeddings for Agentic Insight Retrievalcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
  2. 2025ComoRAG: A Cognitive-Inspired Memory-Organized RAG for Stateful Long Narrative Reasoning
  3. 2026Narrative Knowledge Weaver: Narrative-Centric Retrieval-Augmented Reasoning for Long-Form Text Understanding
  4. 2025EvolvTrip: Enhancing Literary Character Understanding with Temporal Theory-of-Mind Graphs
  5. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels
  6. 2026Narrative World Model: Narratology-Grounded Writer Memory for Long-Form Fiction
  7. 2024Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMs
  8. 2025Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection