mlsanityData to 14 Oct 2026

From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents

arXiv 2610.04575 ↗First posted 3 Oct 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Activation probes can predict safety-relevant properties of language models with high area under the receiver-operating-characteristic curve (AUROC), but deployed agent monitors make thresholded alarm decisions under tight false-alarm budgets. These are different estimands. We introduce an Operational Validity Contract that fixes a monitor's target, observability, identity, timing, intervention unit, comparator, calibration, and cost. We formalize risk at the semantic request or trajectory level: when one task contains repeated alarm opportunities, row-level AUROC and false positive rate do not identify semantic-unit any-alarm risk. A confidence-certified threshold also requires enough independent negative units, a tie-safe rule, and transport to deployment. Across Models Under Pressure, LASR refusal prediction, immutable AgentDojo, and a prospectively protocol-frozen ST-WebAgentBench replication, joint activation-observable monitors reach AUROC 0.957 and 0.935 on the first two benchmarks, yet their locked 5%/10% detection rates are only .642/.742 and .719/.782, respectively. On AgentDojo, the secondary mean-activation rollout monitor reaches AUROC 0.922 but detects none of 38 positive semantic cases at the locked 5% operating point; thresholds intended for 10% false alarms realize 18.7-20.0% on test. Because the test misses its prospectively frozen 40-positive support gate, we label it support-insufficient. On ST-WebAgentBench, activation reaches AUROC .874, but 23 independent calibration negatives cannot identify even a 10% controller; the locked policy abstains rather than reporting its mechanical zero FPR as a success. An exploratory counterexample also lowers full AUROC while improving realized 10% utility. The fail-closed compiler caps MUP and LASR at restricted predictive value and AgentDojo and ST-Web at representation accessibility; no setting reaches alarm-policy validity.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 22 of its 38 references that are in the map. First posted on arXiv 3 Oct 2026.

Builds on

22 of 38 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026PVDetector: Detecting Prompt Injection Attacks on Purpose-Specific LLM Agents through Policy-Violation Concept Analysis14 Jul
  2. 2026When AUC 0.998 Is Not Enough: A Candidate Evaluation Protocol for Hidden-State Probes of Indirect Prompt Injection in Multimodal Computer-Use Agents22 Jun
  3. 2026From Reward-Hack Activations to Agentic Risk States: Context-Calibrated Mechanistic Monitoring in LLM Agents4 Jun
  4. 2026TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling26 May
  5. 2026SLEIGHT-Bench: A Benchmark of Evasion Attacks Against Agent Monitors15 May
  6. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agents9 Oct

Other references · 16

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 876
  2. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  3. 2025Detecting High-Stakes Interactions with Activation Probescited by 53
  4. 2016Neyman-Pearson (NP) classification algorithms and NP receiver operating characteristics (NP-ROC)cited by 23
  5. 2025False Sense of Security: Why Probing-based Malicious Input Detection Fails to Generalizecited by 19
  6. 2026Accurate Failure Prediction in Agents Does Not Imply Effective Failure Preventioncited by 15
  7. 2026From Confident Closing to Silent Failure: Characterizing False Success in LLM Agentscited by 15
  8. 2026Monitoring Emergent Reward Hacking During Generation via Internal Activationscited by 14
  9. 2022Unifying Evaluation of Machine Learning Safety Monitorscited by 10
  10. 2026AutoMonitor-Bench: Evaluating the Reliability of LLM-Based Misbehavior Monitorcited by 10
  11. 2025The Impact of Off-Policy Training Data on Probe Generalisationcited by 9
  12. 2026CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automationcited by 7
4 more
  1. 2026A Structured Approach to Safety Case Construction for AI Systemscited by 6
  2. 2026Do Activation Monitors Survive Model Updates? Benchmarking, Predicting, and Repairing Activation-Monitor Stalenesscited by 5
  3. 2026CP-Agent: A Calibrated Risk-Controlled Agent for Feedback-Driven Competitive Programmingcited by 3
  4. 2026Refusal Before Decoding: Detecting and Exploiting Refusal Signals in Intermediate LLM Activationscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
  2. 2026Prompted to Discriminate: Generalizing Malicious-Input Probes in the Wild
  3. 2026StepShield: When, Not Whether to Intervene on Rogue Agents
  4. 2026Boiling the Frog: A Multi-Turn Benchmark for Agentic Safety
  5. 2026Positive-Unlabeled Learning for Agent Safety False Alarm Auditing
  6. 2026Organising Trajectory Evidence for Language-Model Agent Assurance: Fragments, Methods, and the Residual
  7. 2026Measuring Activation Control in Large Language Models
  8. 2026REDAgentBench: Executable Red Teaming and Faithful Measurement of LLM Agent Systems