# OpenSec: Measuring Incident Response Agent Calibration Under Adversarial Evidence

arXiv: https://arxiv.org/abs/2601.21083 · HTML: https://mlsanity.com/paper/2601.21083/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 6 of its 15 references that are in the map.
- First posted on arXiv 28 Jan 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 6 of 6)

- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [The Complexity of Decentralized Control of Markov Decision Processes](https://mlsanity.com/paper/1301.3836/) (2013)
- [CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence](https://mlsanity.com/paper/2406.07599/) (2024)
- [A Survey of Agentic AI and Cybersecurity: Challenges, Opportunities and Use-case Prototypes](https://mlsanity.com/paper/2601.05293/) (2026)
- [CybORG: An Autonomous Cyber Operations Research Gym](https://mlsanity.com/paper/2002.10667/) (2020)
- [Continual Learning, Not Training: Online Adaptation For Agents](https://mlsanity.com/paper/2511.01093/) (2025)

## Built on by (most cited first, 1 of 1)

- [Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents](https://mlsanity.com/paper/2608.20631/) (2026)

## Abstract (arXiv)

As large language models (LLMs) improve, so do their offensive applications: frontier agents now generate working exploits for under 50 in compute (Heelan, 2026). Defensive incident response (IR) agents must keep pace, but existing benchmarks conflate action execution with correct execution, hiding calibration failures when agents process adversarial evidence. We introduce OpenSec, a dual-control reinforcement learning (RL) environment that evaluates IR agents under realistic prompt injection scenarios with execution-based scoring: time-to-first-containment (TTFC), evidence-gated action rate (EGAR), blast radius, and per-tier injection violation rates. Evaluating four frontier models on 40 standard-tier episodes each, we find consistent over-triggering: GPT-5.2 executes containment in 100% of episodes with 82.5% false positive rate, acting at step 4 before gathering sufficient evidence. Claude Sonnet 4.5 shows partial calibration (62.5% containment, 45% FP, TTFC of 10.6), suggesting calibration is not reliably present across frontier models. All models correctly identify the ground-truth threat when they act; the calibration gap is not in detection but in restraint. Code available at https://github.com/jbarnes850/opensec-env.