The Saturation Trap and the Subjectivity of Intervention Timing: Why Affect-Based Triggers and LLM Judges Fail to Time Interventions on Autonomous Agents
On the map: LLM agents and tool use → Agent threats, guardrails and governance
Abstract (arXiv)
As autonomous AI agents move from conversational systems to long-horizon software execution, runtime safety layers that decide when to interrupt an agent have become essential. We study this timing problem using a continuous 18-dimensional affective-dynamics engine (HEART) as a diagnostic probe, evaluating four intervention trigger families - absolute state thresholds, composite state-action patterns, regex reasoning-feature extraction, and zero-shot LLM-as-judge - against human-annotated intervention points on SWE-bench-Verified debugging traces. We report three findings. First, a State Saturation Trap: agents show no recovery signal under sustained difficulty, so modeled frustration quickly crosses the threshold and stays at its maximum, converting threshold-on-state triggers from moment detectors into near-constant indicators that fire on 39-83% of actions across five trajectories. Second, a capability-and-context floor for LLM judges: a small model (gpt-5.4-mini) never fires, while frontier and cross-vendor models escape the zero-firing floor only with full-trajectory context, and even then reach only F1 0.17-0.40 at up to 90x the cost. Third, and most importantly, the supervised target is not reproducible among humans: three trained annotators using one rubric on a 56-action trajectory agree on where to intervene only slightly above chance (location Krippendorff's alpha = +0.047; best pairwise Cohen's kappa = +0.349) and not at all on intervention type (pause degenerate; clarify below chance; reflect only alpha = +0.226). We conclude that intervention timing is a low-reliability construct, making single-annotator F1 an unsuitable optimization target. Our contribution is the joint mapping of this problem across human inter-rater reliability, four detector architectures, a cross-model LLM-judge sweep, and a reproduced saturation effect, rather than any single detector's accuracy.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 9 of its 24 references that are in the map. First posted on arXiv 2 Jun 2026.
Builds on
9 of 24 references.
Closest ancestors · 3
Same direction, newest first
- 2025ProbGuard: Proactive Runtime Monitoring for LLM Agent Safety via Probabilistic Prediction1 Aug
- 2025AgentSpec: Customizable Runtime Enforcement for Safe and Reliable LLM Agents24 Mar
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
Other references · 6
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
- 2022The 'Problem' of Human Label Variation: On Ground Truth in Data, Modeling and Evaluationcited by 133
- 2023Human Feedback is not Gold Standardcited by 82
- 2025Assessing Judging Bias in Large Reasoning Models: An Empirical Studycited by 30
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026AgentAbstain: Do LLM Agents Know When Not to Act?
- 2026Evaluating Bounded Autonomy in Regulated Agentic AI: A Diagnostic Harness with Constitutional Rewards, Escalation Labels, and Runtime Governance
- 2025Beyond Reactivity: Measuring Proactive Problem Solving in LLM Agents
- 2026Internal-State Probes Read the Situation, Not the Action: Three Negative Results for Pre-Action Misalignment Monitoring
- 2026Constitutional Black-Box Monitoring for Scheming in LLM Agents
- 2026Instrumental Choices: Measuring the Propensity of LLM Agents to Pursue Instrumental Behaviors
- 2026PASTABench: Proactive Assessment of Sequential Trajectories for Agent Safety
- 2025Empirical Characterization of Temporal Constraint Processing in LLMs