# CritBench: A Framework for Evaluating Cybersecurity Capabilities of Large Language Models in IEC 61850 Digital Substation Environments

arXiv: https://arxiv.org/abs/2604.06019 · HTML: https://mlsanity.com/paper/2604.06019/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 37 references that are in the map.
- First posted on arXiv 7 Apr 2026.

## Builds on (most cited first, 26 of 26)

- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [Evaluating Frontier Models for Dangerous Capabilities](https://mlsanity.com/paper/2403.13793/) (2024)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [LLM Agents can Autonomously Hack Websites](https://mlsanity.com/paper/2402.06664/) (2024)
- [AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attacks](https://mlsanity.com/paper/2403.01038/) (2024)
- [Teams of LLM Agents can Exploit Zero-Day Vulnerabilities](https://mlsanity.com/paper/2406.01637/) (2024)
- [NYU CTF Bench: A Scalable Open-Source Benchmark Dataset for Evaluating LLMs in Offensive Security](https://mlsanity.com/paper/2406.05590/) (2024)
- [CYBERSECEVAL 3: Advancing the Evaluation of Cybersecurity Risks and Capabilities in Large Language Models](https://mlsanity.com/paper/2408.01605/) (2024)
- [HackSynth: LLM Agent and Evaluation Framework for Autonomous Penetration Testing](https://mlsanity.com/paper/2412.01778/) (2024)
- [An Empirical Evaluation of LLMs for Solving Offensive Security Challenges](https://mlsanity.com/paper/2402.11814/) (2024)
- [LLMs as Hackers: Autonomous Linux Privilege Escalation Attacks](https://mlsanity.com/paper/2310.11409/) (2023)
- [AutoPenBench: Benchmarking Generative Agents for Penetration Testing](https://mlsanity.com/paper/2410.03225/) (2024)
- [D-CIPHER: Dynamic Collaborative Intelligent Multi-Agent System with Planner and Heterogeneous Executors for Offensive Security](https://mlsanity.com/paper/2502.10931/) (2025)
- [A Framework for Evaluating Emerging Cyberattack Capabilities of AI](https://mlsanity.com/paper/2503.11917/) (2025)
- [Benchmarking Practices in LLM-driven Offensive Security: Testbeds, Metrics, and Experiment Design](https://mlsanity.com/paper/2504.10112/) (2025)
- [ChatGPT and Other Large Language Models for Cybersecurity of Smart Grid Applications](https://mlsanity.com/paper/2311.05462/) (2023)
- [Catastrophic Cyber Capabilities Benchmark (3CB): Robustly Evaluating LLM Agent Cyber Offense Capabilities](https://mlsanity.com/paper/2410.09114/) (2024)
- [OCCULT: Evaluating Large Language Models for Offensive Cyber Operation Capabilities](https://mlsanity.com/paper/2502.15797/) (2025)
- [Cybersecurity AI Benchmark (CAIBench): A Meta-Benchmark for Evaluating Cybersecurity AI Agents](https://mlsanity.com/paper/2510.24317/) (2025)
- [RapidPen: Fully Automated IP-to-Shell Penetration Testing with LLM-based Agents](https://mlsanity.com/paper/2502.16730/) (2025)
- [On the Surprising Efficacy of LLMs for Penetration-Testing](https://mlsanity.com/paper/2507.00829/) (2025)
- [Vulnerability of Machine Learning Approaches Applied in IoT-based Smart Grid: A Review](https://mlsanity.com/paper/2308.15736/) (2023)
- [Got Root? A Linux Priv-Esc Benchmark](https://mlsanity.com/paper/2405.02106/) (2024)
- [AttackLLM: LLM-based Attack Pattern Generation for an Industrial Control System](https://mlsanity.com/paper/2504.04187/) (2025)
- [LLMPot: Dynamically Configured LLM-based Honeypot for Industrial Protocol and Physical Process Emulation](https://mlsanity.com/paper/2405.05999/) (2024)
- [Cybersecurity AI: Humanoid Robots as Attack Vectors](https://mlsanity.com/paper/2509.14139/) (2025)

## Built on by (most cited first, 1 of 1)

- [NRT-Bench: Benchmarking Multi-Turn Red-Teaming of LLM Operator Agents in Safety-Critical Control Rooms](https://mlsanity.com/paper/2606.20408/) (2026)

## Abstract (arXiv)

The advancement of Large Language Models (LLMs) has raised concerns regarding their dual-use potential in cybersecurity. Existing evaluation frameworks overwhelmingly focus on Information Technology (IT) environments, failing to capture the constraints, and specialized protocols of Operational Technology (OT). To address this gap, we introduce CritBench, a novel framework designed to evaluate the cybersecurity capabilities of LLM agents within IEC 61850 Digital Substation environments. We assess five state-of-the-art models, including OpenAI's GPT-5 suite and open-weight models, across a corpus of 81 domain-specific tasks spanning static configuration analysis, network traffic reconnaissance, and live virtual machine interaction. To facilitate industrial protocol interaction, we develop a domain-specific tool scaffold. Our empirical results show that agents reliably execute static structured-file analysis and single-tool network enumeration, but their performance degrades on dynamic tasks. Despite demonstrating explicit, internalized knowledge of the IEC 61850 standards terminology, current models struggle with the persistent sequential reasoning and state tracking required to manipulate live systems without specialized tools. Equipping agents with our domain-specific tool scaffold significantly mitigates this operational bottleneck. Code and evaluation scripts are available at: https://github.com/GKeppler/CritBench