# DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

arXiv: https://arxiv.org/abs/2608.03591 · HTML: https://mlsanity.com/paper/2608.03591/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 20 of its 58 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 20 of 20)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threats](https://mlsanity.com/paper/2001.01525/) (2020)
- [Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenance](https://mlsanity.com/paper/2308.05034/) (2023)
- [SLEUTH: Real-time Attack Scenario Reconstruction from COTS Audit Data](https://mlsanity.com/paper/1801.02062/) (2018)
- [POIROT: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Hunting](https://mlsanity.com/paper/1910.00056/) (2019)
- [Benchmarking Large Language Models for Log Analysis, Security, and Interpretation](https://mlsanity.com/paper/2311.14519/) (2023)
- [ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation](https://mlsanity.com/paper/2507.14201/) (2025)
- [CyberRAG: An Agentic RAG cyber attack classification and reporting tool](https://mlsanity.com/paper/2507.02424/) (2025)
- [OCR-APT: Reconstructing APT Stories from Audit Logs using Subgraph Anomaly Detection and LLMs](https://mlsanity.com/paper/2510.15188/) (2025)
- [AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understanding](https://mlsanity.com/paper/2503.03170/) (2025)
- [Before You Hand Over the Wheel: Evaluating LLMs for Security Incident Analysis](https://mlsanity.com/paper/2603.06422/) (2026)
- [Retrieval-Augmented LLMs for Security Incident Analysis](https://mlsanity.com/paper/2603.18196/) (2026)
- [Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps](https://mlsanity.com/paper/2604.19533/) (2026)
- [OMNISEC: LLM-Driven Provenance-based Intrusion Detection via Retrieval-Augmented Behavior Prompting](https://mlsanity.com/paper/2503.03108/) (2025)
- [An Empirical Study of Observability Limits in Advanced Software Supply Chain Attacks](https://mlsanity.com/paper/2603.16694/) (2026)
- [Beyond RAG for Cyber Threat Intelligence: A Systematic Evaluation of Graph-Based and Agentic Retrieval](https://mlsanity.com/paper/2604.11419/) (2026)
- [SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents](https://mlsanity.com/paper/2604.12040/) (2026)
- [Benchmarking and Exploring the Capabilities of LLMs for Attack Investigations](https://mlsanity.com/paper/2606.10281/) (2026)
- [FuseChain: Runtime Evidence Reconstruction for Software Supply-Chain Attacks](https://mlsanity.com/paper/2606.15811/) (2026)

## Built on by (most cited first, 3 of 3)

- [Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigations](https://mlsanity.com/paper/2609.30345/) (2026)
- [CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenance](https://mlsanity.com/paper/2609.32424/) (2026)
- [APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetry](https://mlsanity.com/paper/2609.38954/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions. However, existing benchmarks mainly evaluate final outputs or aggregate accuracy, providing limited insight into how errors arise and propagate across intermediate reasoning stages. We present DiagChain, a diagnostic benchmark for evidence-grounded attack chain reconstruction that enables stage-wise evaluation of LLM agents. DiagChain includes MAIN-69, a suite of 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths. It further introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG), which couples evidence retrieval with an evolving structured representation of the reconstructed chain. Five complementary metrics are introduced to assess distinct stages of the reconstruction process and support systematic failure diagnosis. Based on evaluations using 6 LLMs, DiagChain reveals that even the strongest configuration succeeds on only 39.6% of the 849 reference steps in MAIN-69. Our analysis further shows that smaller models struggle with the more basic task of incorporating retrieved evidence into their outputs, whereas larger models can proceed to later steps, where correctly ordering that evidence becomes the main bottleneck. These results validate the importance of diagnostic evaluation beyond end-to-end accuracy and provide actionable insights for improving evidence-grounded cybersecurity agents.