mlsanityData to 7 Oct 2026

DiagChain: A Diagnostic Benchmark for Evaluating LLM Agents on Evidence-Grounded Attack Chain Reconstruction

arXiv 2608.03591 ↗First posted 4 Aug 2026cs.CR cs.AIcited by 3

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large Language Model (LLM) agents offer a promising approach to attack chain reconstruction by retrieving and interpreting heterogeneous telemetry to infer ordered attacker actions. However, existing benchmarks mainly evaluate final outputs or aggregate accuracy, providing limited insight into how errors arise and propagate across intermediate reasoning stages. We present DiagChain, a diagnostic benchmark for evidence-grounded attack chain reconstruction that enables stage-wise evaluation of LLM agents. DiagChain includes MAIN-69, a suite of 69 scenarios spanning multiple operating systems, evidence noise levels, and chain lengths. It further introduces Evidence-Centric Retrieval-Augmented Generation (ECRAG), which couples evidence retrieval with an evolving structured representation of the reconstructed chain. Five complementary metrics are introduced to assess distinct stages of the reconstruction process and support systematic failure diagnosis. Based on evaluations using 6 LLMs, DiagChain reveals that even the strongest configuration succeeds on only 39.6% of the 849 reference steps in MAIN-69. Our analysis further shows that smaller models struggle with the more basic task of incorporating retrieved evidence into their outputs, whereas larger models can proceed to later steps, where correctly ordering that evidence becomes the main bottleneck. These results validate the importance of diagnostic evaluation beyond end-to-end accuracy and provide actionable insights for improving evidence-grounded cybersecurity agents.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 20 of its 58 references that are in the map. First posted on arXiv 4 Aug 2026.

Builds on

20 of 58 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Cyber Defense Benchmark: Agentic Threat Hunting Evaluation for LLMs in SecOps21 Apr
  2. 2026SIR-Bench: Evaluating Investigation Depth in Security Incident Response Agents13 Apr
  3. 2026Beyond RAG for Cyber Threat Intelligence: A Systematic Evaluation of Graph-Based and Agentic Retrieval13 Apr
  4. 2025ExCyTIn-Bench: Evaluating LLM agents on Cyber Threat Investigation14 Jul
  5. 2025CyberRAG: An Agentic RAG cyber attack classification and reporting tool3 Jul

Other references · 15

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  3. 2020UNICORN: Runtime Provenance-Based Detector for Advanced Persistent Threatscited by 53
  4. 2023Kairos: Practical Intrusion Detection and Investigation using Whole-system Provenancecited by 30
  5. 2018SLEUTH: Real-time Attack Scenario Reconstruction from COTS Audit Datacited by 28
  6. 2019POIROT: Aligning Attack Behavior with Kernel Audit Records for Cyber Threat Huntingcited by 27
  7. 2023Benchmarking Large Language Models for Log Analysis, Security, and Interpretationcited by 19
  8. 2025OCR-APT: Reconstructing APT Stories from Audit Logs using Subgraph Anomaly Detection and LLMscited by 6
  9. 2026Before You Hand Over the Wheel: Evaluating LLMs for Security Incident Analysiscited by 4
  10. 2026Retrieval-Augmented LLMs for Security Incident Analysiscited by 4
  11. 2025OMNISEC: LLM-Driven Provenance-based Intrusion Detection via Retrieval-Augmented Behavior Promptingcited by 3
  12. 2025AttackSeqBench: Benchmarking the Capabilities of LLMs for Attack Sequences Understandingcited by 3
3 more
  1. 2026An Empirical Study of Observability Limits in Advanced Software Supply Chain Attackscited by 3
  2. 2026Benchmarking and Exploring the Capabilities of LLMs for Attack Investigationscited by 2
  3. 2026FuseChain: Runtime Evidence Reconstruction for Software Supply-Chain Attackscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Coding Agents Aren't Enough! Evaluating an Enterprise Security Brain for Agentic Cloud Investigationscited by 0
  2. 2026CyberClear: A Benchmark for LLM Agent Systems on APT Attack Chain Provenancecited by 0
  3. 2026APTInvestBench: Evaluating Autonomous APT Investigation under Varying Telemetrycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems
  2. 2026Hierarchical Agentic Incident Response with Digital-Twin-Validated Attack Inference
  3. 2026AgentDrift: A Step-Labeled Benchmark of Injection-Hijacked LLM Agent Trajectories
  4. 2025Efficient LLM Safety Evaluation through Multi-Agent Debate
  5. 2026PropGuard: Safeguarding LLM-MAS via Propagation-Aware Exploration and Remediation
  6. 2025Guided Reasoning in LLM-Driven Penetration Testing Using Structured Attack Trees
  7. 2026From Investigation Failures to Reliable SOC Agents: Understanding and Improving LLM-Based Alert Triage
  8. 2026On the Effectiveness of Kernel-Level Evidence for Agent Security