# Calibrated Decision Models for Autonomous Penetration-Testing Harnesses: JEV and Laya as System One Decision Layers for LLM-Driven Pentest Agents

arXiv: https://arxiv.org/abs/2609.28940 · HTML: https://mlsanity.com/paper/2609.28940/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 39 references that are in the map.
- First posted on arXiv 24 Sep 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 15 of 15)

- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2307.15217/) (2023)
- [Cybench: A Framework for Evaluating Cybersecurity Capabilities and Risks of Language Models](https://mlsanity.com/paper/2408.08926/) (2024)
- [LLM Agents can Autonomously Exploit One-day Vulnerabilities](https://mlsanity.com/paper/2404.08144/) (2024)
- [JEV-as-a-Judge: Accept When Confident, Escalate When Unsure](https://mlsanity.com/paper/2609.26550/) (2026)
- [Multi-Agent Penetration Testing AI for the Web](https://mlsanity.com/paper/2508.20816/) (2025)
- [Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents](https://mlsanity.com/paper/2609.23986/) (2026)
- [Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)](https://mlsanity.com/paper/2609.24052/) (2026)
- [Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences](https://mlsanity.com/paper/2609.24965/) (2026)
- [REFLEX with Jev for Efficient Selective Control in LLM Agents](https://mlsanity.com/paper/2609.26532/) (2026)
- [Cybersecurity AI: The Dangerous Gap Between Automation and Autonomy](https://mlsanity.com/paper/2506.23592/) (2025)
- [Penetration Testing of Agentic AI: A Comparative Security Analysis Across Models and Frameworks](https://mlsanity.com/paper/2512.14860/) (2025)
- [Baselines Before Architecture: Evaluating Coding Agents for Autonomous Penetration Testing](https://mlsanity.com/paper/2607.13085/) (2026)
- [The Role of AI in Modern Penetration Testing](https://mlsanity.com/paper/2512.12326/) (2025)
- [From Capability to Assurance in Autonomous Penetration-Testing Harnesses: A Framework and Reference Implementation](https://mlsanity.com/paper/2609.22664/) (2026)

## Built on by (most cited first, 4 of 4)

- [Typed Decision Models: An Early Evidence Audit and Evaluation Checklist](https://mlsanity.com/paper/2609.32160/) (2026)
- [Jev in Medicine: A Benchmark Evaluation](https://mlsanity.com/paper/2609.34024/) (2026)
- [From Retrieval to Typed Decisions: Calibrated System One Models from Biomedical Sentence Encoders](https://mlsanity.com/paper/2610.02486/) (2026)
- [Token-Efficient Multi-Agent Collaboration via System One-Guided Computational Division of Labor](https://mlsanity.com/paper/2610.08155/) (2026)

## Abstract (arXiv)

Autonomous penetration-testing harnesses use large language models (LLMs) for reconnaissance, exploitation, and reporting, but often rely on those same models to confirm findings, grade severity, and select agents. This can lead to false positives, inflated severity, and wasted compute. We examine how System One decision models, lightweight non-generative classifiers that return typed, calibrated verdicts, can support these decisions. We make five contributions. First, we define four decision points: finding adjudication, severity recalibration, agent pruning, and confirmation loops. Second, we present an exploratory NeuroSploit case study comparing one run with TypeSafe System One (Jev) and one without it against a web target containing 13 vulnerabilities. Differences in severity distribution, runtime, and grading by exposed data type motivate the architecture but do not establish statistical significance. Third, we review published specifications for Jev, Jev-Ultrafast, and the open-source Laya without assuming that results from other benchmarks transfer to penetration testing. Fourth, we discuss RLHF, RLAIF, RLCD, and RLHV as training approaches and their implications for trust in security decisions. Finally, we propose Rave, a domain-adapted System One model, and outline its training data, evaluation protocol, and potential effect on harness assurance.