# Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

arXiv: https://arxiv.org/abs/2609.33401 · HTML: https://mlsanity.com/paper/2609.33401/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 39 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 23 of 23)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs](https://mlsanity.com/paper/2406.18495/) (2024)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents](https://mlsanity.com/paper/2503.18666/) (2025)
- [AgentDyn: Are Your Agent Security Defenses Deployable in Real-World Dynamic Environments?](https://mlsanity.com/paper/2602.03117/) (2026)
- [JEV-as-a-Judge: Accept When Confident, Escalate When Unsure](https://mlsanity.com/paper/2609.26550/) (2026)
- [ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction](https://mlsanity.com/paper/2508.00500/) (2025)
- [Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It](https://mlsanity.com/paper/2609.26758/) (2026)
- [WAInjectBench: Benchmarking Prompt Injection Detections for Web Agents](https://mlsanity.com/paper/2510.01354/) (2025)
- [REFLEX with Jev for Efficient Selective Control in LLM Agents](https://mlsanity.com/paper/2609.26532/) (2026)
- [Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding](https://mlsanity.com/paper/2609.27678/) (2026)
- [Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions](https://mlsanity.com/paper/2609.28613/) (2026)
- [Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures](https://mlsanity.com/paper/2609.29429/) (2026)
- [Jev in the Wild: A Data-Driven Analysis of the Jev Model's Functionality, Applications and Ecosystem](https://mlsanity.com/paper/2609.30216/) (2026)
- [Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places](https://mlsanity.com/paper/2609.29769/) (2026)
- [JevOut: Natural Context Can Flip Decision Models](https://mlsanity.com/paper/2609.30243/) (2026)
- [Confidently Wrong: Severity-Aware Calibration of Prompt-Injection Detectors under Attack Shift](https://mlsanity.com/paper/2606.22659/) (2026)
- [PIDS-Bench: Evaluating Prompt-Injection Detectors Under Over-Defense, Obfuscation, and Distribution Shift](https://mlsanity.com/paper/2609.15017/) (2026)

## Abstract (arXiv)

Agentic software connects language models to tools that modify files and interact with external services. Developers use model-based security checks to screen external content and user requests before agents act. System One models answer developer-defined questions with probabilities over predefined answers such as safe or unsafe, but classification accuracy alone does not establish whether these probabilities support reliable automation. We evaluate Jev, Laya, Decider, and Bespoke Nimble against specialized classifiers and language-model judges across prompt-injection detection, interaction-risk judgment, and harmful-request screening, examining decision accuracy, calibration, and selective automation. (1) High overall accuracy and low average calibration error can conceal attacks classified as safe with high confidence within particular groups. Developers should test candidate models on the intended security task and examine errors within relevant attack groups. (2) Policies selected under strict miss limits allow few test inputs automatically, and choosing allow and block thresholds separately increases automation mainly through additional blocks. Developers should verify limits on missed unsafe inputs and blocked benign inputs on independent data, and report the allowed and blocked fractions separately. (3) Comparing predictions on the same inputs shows that a language-model judge can detect unsafe inputs missed by a System One model, but can also repeat the System One model's confident mistakes and falsely flag benign inputs. Developers should test which missed unsafe inputs their review rule forwards, then measure the review model's misses and benign false alarms on those inputs.