mlsanityData to 7 Oct 2026

DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels

arXiv 2409.02465 ↗First posted 4 Sep 2024cs.CLcited by 21

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Recently, significant efforts have been devoted to enhancing the long-context capabilities of Large Language Models (LLMs), particularly in long-context reasoning. To facilitate this research, we propose DetectiveQA, a dataset specifically designed for narrative reasoning within long contexts. We leverage detective novels, averaging over 100k tokens, to create a dataset containing 1200 human-annotated questions in both Chinese and English, each paired with corresponding reference reasoning steps. Furthermore, we introduce a step-wise reasoning metric, which enhances the evaluation of LLMs' reasoning processes. We validate our approach and evaluate the mainstream LLMs, including GPT-4, Claude, and LLaMA, revealing persistent long-context reasoning challenges and demonstrating their evidence-retrieval challenges. Our findings offer valuable insights into the study of long-context reasoning and lay the base for more rigorous evaluations.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 33 of its 46 references that are in the map. First posted on arXiv 4 Sep 2024.

Builds on

33 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack14 Jun
  2. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey2 Apr
  3. 2022True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-420 Dec

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  6. 2023Qwen Technical Reportcited by 3,273
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  10. 2023MetaGPT: Meta Programming for A Multi-Agent Collaborative Frameworkcited by 1,761
  11. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  12. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
18 more
  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  3. 2023C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Modelscited by 694
  4. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  5. 2019ELI5: Long Form Question Answeringcited by 567
  6. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  7. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  8. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  9. 2021QuALITY: Question Answering with Long Input Texts, Yes!cited by 204
  10. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  11. 2023ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understandingcited by 175
  12. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  13. 2023Large Language Models for Robotics: A Surveycited by 132
  14. 2023Retrieval meets Long Context Large Language Modelscited by 114
  15. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  16. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  17. 2024In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Misscited by 36
  18. 2022CAB: Comprehensive Attention Benchmarking on Long Sequence Modelingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  2. 2025ComoRAG: A Cognitive-Inspired Memory-Organized RAG for Stateful Long Narrative Reasoningcited by 13
  3. 2025WildLong: Synthesizing Realistic Long-Context Instruction Data at Scalecited by 9
  4. 2025SitEmb-v1.5: Improved Context-Aware Dense Retrieval for Semantic Association and Long Story Comprehensioncited by 8
  5. 2026AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rolloutscited by 8
  6. 2025A Controllable Examination for Long-Context Language Modelscited by 7
  7. 2026Query-focused and Memory-aware Reranker for Long Context Processingcited by 6
  8. 2025Mindscape-Aware Retrieval Augmented Generation for Improved Long Context Understandingcited by 5
  9. 2026Architectural Design, Not Only Model Intelligence, Governs Multi-Agent LLM Performancecited by 5
  10. 2026Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMscited by 5
  11. 2025PRISON: Unmasking the Criminal Potential of Large Language Modelscited by 2
  12. 2025Evaluation Hallucination in Multi-Round Incomplete Information Lateral-Driven Reasoning Taskscited by 1
9 more
  1. 2026MiA-Signature: Approximating Global Activation for Long-Context Understandingcited by 1
  2. 2026When Your Agent Opens the Chat App: Agent-Controlled Search over Raw Chat Logs Rivals Structured Memorycited by 1
  3. 2026Trace Only What You Need: Structure-Aware On-Demand Hypergraph Memory for Long-Document Question Answeringcited by 0
  4. 2026Storyline Trees: Hierarchical Representations for Long-Form Narrativescited by 0
  5. 2026RECON: Benchmarking Agent Memory for Compositional Reasoning over Long Contextscited by 0
  6. 2026Towards High-Level Semantic Intelligencecited by 0
  7. 2026Beyond Linear Context: Graph-Guided Evidence Navigation for Long-Novel Reasoning with a Local 9B Language Modelcited by 0
  8. 2026Contract Memory Compiler: Resolve, Then Traversecited by 0
  9. 2026LSTMem: Hierarchical Long Short-Term Online Memory for Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024NovelQA: Benchmarking Question Answering on Documents Exceeding 200K Tokens
  2. 2025NovelHopQA: Diagnosing Multi-Hop Reasoning Failures in Long Narrative Contexts
  3. 2026ClueWeaver: Reward-Guided Dual-Agent Evidence Reasoning for Compact LLMs on Literary Long Narratives
  4. 2025SagaScale: A Realistic, Scalable, and High-Quality Long-Context Benchmark Built from Full-Length Novels
  5. 2025LongReason: A Synthetic Long-Context Reasoning Benchmark via Context Expansion
  6. 2025SCoRE: Benchmarking Long-Chain Reasoning in Commonsense Scenarios
  7. 2025TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Games
  8. 2026π2: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models