# LLM-as-an-Investigator: Evidence-First Reasoning for Robust Interactive Problem Diagnosis

arXiv: https://arxiv.org/abs/2606.13220 · HTML: https://mlsanity.com/paper/2606.13220/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 25 of its 25 references that are in the map.
- First posted on arXiv 11 Jun 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 25 of 25)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Towards Understanding Sycophancy in Language Models](https://mlsanity.com/paper/2310.13548/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications](https://mlsanity.com/paper/2402.07927/) (2024)
- [HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2305.11747/) (2023)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [SycEval: Evaluating LLM Sycophancy](https://mlsanity.com/paper/2502.08177/) (2025)
- [CLAM: Selective Clarification for Ambiguous Questions with Generative Language Models](https://mlsanity.com/paper/2212.07769/) (2022)
- [When Large Language Models contradict humans? Large Language Models' Sycophantic Behaviour](https://mlsanity.com/paper/2311.09410/) (2023)
- [Tree of Clarifications: Answering Ambiguous Questions with Retrieval-Augmented Large Language Models](https://mlsanity.com/paper/2310.14696/) (2023)
- [An In-depth Survey of Large Language Model-based Artificial Intelligence Agents](https://mlsanity.com/paper/2309.14365/) (2023)
- [Invisible Saboteurs: Sycophantic LLMs Mislead Novices in Problem-Solving Tasks](https://mlsanity.com/paper/2510.03667/) (2025)
- [Good Arguments Against the People Pleasers: How Reasoning Mitigates (Yet Masks) LLM Sycophancy](https://mlsanity.com/paper/2603.16643/) (2026)
- [Iterative Resolution of Prompt Ambiguities Using a Progressive Cutting-Search Approach](https://mlsanity.com/paper/2505.02952/) (2025)
- [Generative AI for Managerial Decision-Making under Ambiguity and Sycophancy](https://mlsanity.com/paper/2603.03970/) (2026)

## Built on by (most cited first, 1 of 1)

- [Not Until the Evidence Says So: Teaching LLM Investigators When to Close a Case](https://mlsanity.com/paper/2610.03190/) (2026)

## Abstract (arXiv)

Large language models (LLMs) are used for technical troubleshooting, where problem descriptions are often incomplete and multiple causes can explain the same symptoms. Effective diagnosis requires acquiring discriminative evidence and selecting the explanation best supported by it. Yet reasoning or interaction alone may not prevent premature commitment to an incorrect hypothesis. We introduce LLM-as-an-Investigator, an evidence-first methodology for interactive problem diagnosis that reduces premature commitment while distinguishing competing causes. We implement it through the Solution Investigator Agent (SIA), which orchestrates three complementary reasoning behaviours through a shared investigation state: a Diagnostic Reasoner that conducts and reassesses the diagnostic interview, a Hypothesis Investigator that maintains and updates competing hypotheses, and a Counterfactual Investigator that challenges current explanations and guides discriminative follow-up questions. Final selection uses cited dialogue evidence and a separate verification step, while explicit hypothesis and evidence records make the diagnosis traceable and preserve unresolved alternatives. We construct a structured corpus from 303 technical forum threads across mechanical, electrical, and hydraulic domains. A source-grounded pipeline separates the initial problem, pre-solution clarifications, and reported outcome, while a user simulator answers diagnostic questions without access to the reference solution. We compare SIA with direct, reasoning-based, interactive, and search-based baselines, including adaptations of ReAct and Tree of Thoughts. Evaluation distinguishes final diagnostic quality from hypothesis coverage and oracle potential, while ablations and paired stress tests assess component contributions and robustness to plausible but misleading user hypotheses.