mlsanityData to 7 Oct 2026

True Detective: A Deep Abductive Reasoning Benchmark Undoable for GPT-3 and Challenging for GPT-4

arXiv 2212.10114 ↗First posted 20 Dec 2022cs.CLcited by 19

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) have demonstrated solid zero-shot reasoning capabilities, which is reflected in their performance on the current test tasks. This calls for a more challenging benchmark requiring highly advanced reasoning ability to be solved. In this paper, we introduce such a benchmark, consisting of 191 long-form (1200 words on average) mystery narratives constructed as detective puzzles. Puzzles are sourced from the "5 Minute Mystery" platform and include a multiple-choice question for evaluation. Only 47% of humans solve a puzzle successfully on average, while the best human solvers achieve over 80% success rate. We show that GPT-3 models barely outperform random on this benchmark (with 28% accuracy) while state-of-the-art GPT-4 solves only 38% of puzzles. This indicates that there is still a significant gap in the deep reasoning abilities of LLMs and humans and highlights the need for further research in this area. Our work introduces a challenging benchmark for future studies on reasoning in language models and contributes to a better understanding of the limits of LLMs' abilities.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 13 of its 23 references that are in the map. First posted on arXiv 20 Dec 2022.

Builds on

13 of 23 references.

References · 13

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2016A Corpus and Evaluation Framework for Deeper Understanding of Commonsense Storiescited by 413
  12. 2018SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inferencecited by 333
1 more
  1. 2019Abductive Commonsense Reasoningcited by 300

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 3; 2025: 10; 2026: 5.

19 papers.

Sort:

By area:

  1. 2023Unleashing the potential of prompt engineering for large language modelscited by 72
  2. 2025Which of These Best Describes Multiple Choice Evaluation with LLMs? A) Forced B) Flawed C) Fixable D) All of the Abovecited by 51
  3. 2024Puzzle Solving using Reasoning of Large Language Models: A Surveycited by 44
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  5. 2024DetectiveQA: Evaluating Long-Context Reasoning on Detective Novelscited by 21
  6. 2025LogiDynamics: Unraveling the Dynamics of Inductive, Abductive and Deductive Logical Inferences in LLM Reasoningcited by 12
  7. 2025Controllable Logical Hypothesis Generation for Abductive Reasoning in Knowledge Graphscited by 11
  8. 2025Unifying Deductive and Abductive Reasoning in Knowledge Graphs with Masked Diffusion Modelcited by 8
  9. 2026Soft Contamination Means Benchmarks Test Shallow Generalizationcited by 8
  10. 2025Evaluating the Logical Reasoning Abilities of Large Reasoning Modelscited by 7
  11. 2026Wiring the 'Why': A Unified Taxonomy and Survey of Abductive Reasoning in LLMscited by 5
  12. 2024Unveiling Narrative Reasoning Limits of Large Language Models with Trope in Movie Synopsescited by 4
7 more
  1. 2025The Jumping Reasoning Curve? Tracking the Evolution of Reasoning Performance in GPT-[n] and o-[n] Models on Multimodal Puzzlescited by 4
  2. 2025TurnaboutLLM: A Deductive Reasoning Benchmark from Detective Gamescited by 4
  3. 2025Socrates or Smartypants: Testing Logic Reasoning Capabilities of Large Language Models with Logic Programming-based Test Oraclescited by 2
  4. 2026Self-Awareness before Action: Mitigating Logical Inertia via Proactive Cognitive Awarenesscited by 1
  5. 2025The Challenge and Reward of Fair Play in Narrative: A Computational Approachcited by 0
  6. 2026HypoAgent: An Agentic Framework for Interactive Abductive Hypothesis Generation over Knowledge Graphscited by 0
  7. 2026EVAR: Evidence-Validated Hypothesis Admission for Budget-Aware Narrative Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Watson & Holmes: A Naturalistic Benchmark for Comparing Human and LLM Reasoning
  2. 2024REBUS: A Robust Evaluation Benchmark of Understanding Symbols
  3. 2025RiddleBench: A New Generative Reasoning Benchmark for LLMs
  4. 2025From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information?
  5. 2025Do You Get the Hint? Benchmarking LLMs on the Board Game Concept
  6. 2025ReasoningWeekly: A General Knowledge and Verbal Reasoning Challenge for Large Language Models
  7. 2025PuzzleWorld: A Benchmark for Multimodal, Open-Ended Reasoning in Puzzlehunts
  8. 2025MultiZebraLogic: A Multilingual Logical Reasoning Benchmark