# Piecing Together Clues: A Benchmark for Evaluating the Detective Skills of Large Language Models

arXiv: https://arxiv.org/abs/2307.05113 · HTML: https://mlsanity.com/paper/2307.05113/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 11 Jul 2023.

## Builds on (most cited first, 20 of 20)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [HellaSwag: Can a Machine Really Finish Your Sentence?](https://mlsanity.com/paper/1905.07830/) (2019)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [FEVER: a large-scale dataset for Fact Extraction and VERification](https://mlsanity.com/paper/1803.05355/) (2018)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies](https://mlsanity.com/paper/2101.02235/) (2021)
- [Complexity-Based Prompting for Multi-Step Reasoning](https://mlsanity.com/paper/2210.00720/) (2022)
- [Abductive Commonsense Reasoning](https://mlsanity.com/paper/1908.05739/) (2019)
- [Unified Structure Generation for Universal Information Extraction](https://mlsanity.com/paper/2203.12277/) (2022)
- [ReClor: A Reading Comprehension Dataset Requiring Logical Reasoning](https://mlsanity.com/paper/2002.04326/) (2020)
- [FEVEROUS: Fact Extraction and VERification Over Unstructured and Structured information](https://mlsanity.com/paper/2106.05707/) (2021)
- [Open-Domain Question Answering Goes Conversational via Question Rewriting](https://mlsanity.com/paper/2010.04898/) (2020)
- [TopiOCQA: Open-domain Conversational Question Answering with Topic Switching](https://mlsanity.com/paper/2110.00768/) (2021)
- [Xiezhi: An Ever-Updating Benchmark for Holistic Domain Knowledge Evaluation](https://mlsanity.com/paper/2306.05783/) (2023)
- [INSCIT: Information-Seeking Conversations with Mixed-Initiative Interactions](https://mlsanity.com/paper/2207.00746/) (2022)
- [UQuAD1.0: Development of an Urdu Question Answering Training Data for Machine Reading Comprehension](https://mlsanity.com/paper/2111.01543/) (2021)
- [Contextual embedding and model weighting by fusing domain knowledge on Biomedical Question Answering](https://mlsanity.com/paper/2206.12866/) (2022)

## Built on by (most cited first, 2 of 2)

- [Step-by-Step Reasoning to Solve Grid Puzzles: Where do LLMs Falter?](https://mlsanity.com/paper/2407.14790/) (2024)
- [CaughtCheating: Is Your MLLM a Good Cheating Detective? Exploring the Boundary of Visual Perception and Reasoning](https://mlsanity.com/paper/2507.00045/) (2025)

## Abstract (arXiv)

Detectives frequently engage in information detection and reasoning simultaneously when making decisions across various cases, especially when confronted with a vast amount of information. With the rapid development of large language models~(LLMs), evaluating how these models identify key information and reason to solve questions becomes increasingly relevant. We introduces the DetectBench, a reading comprehension dataset designed to assess a model's ability to jointly ability in key information detection and multi-hop reasoning when facing complex and implicit information. The DetectBench comprises 3,928 questions, each paired with a paragraph averaging 190 tokens in length. To enhance model's detective skills, we propose the Detective Thinking Framework. These methods encourage models to identify all possible clues within the context before reasoning. Our experiments reveal that existing models perform poorly in both information detection and multi-hop reasoning. However, the Detective Thinking Framework approach alleviates this issue.