CURA: Certified Runtime Alarms for Computer-Use Agents
Not on the map. Why?
Abstract (arXiv)
Self-report is the cheapest oversight channel a deployer has, and on capable computer-use agents (CUAs) it fails precisely where oversight matters. On 361 OSWorld tasks our pipeline, a read-only feasibility gate, a planner, and a GUI executor, reaches a mean task score of 82.9, above the 72.4 human reference, yet 64 of its 71 failures (90%) end with a success claim, 61 acknowledging no blocker, and the explicit failure affordance is never used in roughly 9,100 calls. We introduce CURA (Certified Runtime Alarms for Computer-Use Agents), an external monitor that reads only harness-visible telemetry, with no model internals, extra LLM calls, or prompt changes, and turns the running trajectory into a sequential test with certified false-alarm control. At alpha = 0.10 its CUSUM alarm detects 42.3% of failures a median of 31 steps before termination at a realized false-alarm rate of 0.066, and risk is partly resolvable before the first action (gate probe, 0.69 AUROC). Retrospectively the composite reaches 0.828 AUROC (fold-internal floor 0.802), but its margin over a total-token baseline is not significant (Delta = +0.026, p = 0.101); the separation is online, where CURA recalls more at matched certified budgets: 0.41 versus 0.34 at alpha = 0.10, 0.56 versus 0.38 at alpha = 0.20. Alarm-gated mid-execution oversight recovers 23 of 70 failures while spending a frontier overseer on 38, giving a deployable cascade at mean score 86.8 and 84.5% full-solve (305 of 361). The certificate bounds false alarms only. We also report where behavioral monitoring is uninformative.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 19 of its 117 references that are in the map. First posted on arXiv 28 Aug 2026.
Builds on
19 of 117 references.
References · 19
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2017Selective Classification for Deep Neural Networkscited by 852
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
- 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
- 2024Frontier Models are Capable of In-context Schemingcited by 287
- 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
- 2022Game-theoretic statistics and safe anytime-valid inferencecited by 155
- 2025GTA1: GUI Test-time Scaling Agentcited by 106
- 2021Sample-Efficient Safety Assurances using Conformal Predictioncited by 41
- 2019Testing randomnesscited by 34
7 more
- 2022Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehiclescited by 11
- 2026TRACER: Trajectory Risk Aggregation for Critical Episodes in Agentic Reasoningcited by 11
- 2026VLM Judges Can Rank but Cannot Score: Task-Dependent Uncertainty in Multimodal Evaluationcited by 8
- 2025Learnable Conformal Prediction with Context-Aware Nonconformity Functions for Robotic Planning and Perceptioncited by 7
- 2025Calibrated Decomposition of Aleatoric and Epistemic Uncertainty in Deep Features for Inference-Time Adaptationcited by 6
- 2026TRIAGE: Type-Routed Interventions via Aleatoric-Epistemic Gated Estimation in Robotic Manipulation and Adaptive Perception -- Don't Treat All Uncertainty the Samecited by 3
- 2026Uncertainty Quantification for Computer-Use Agents: A Benchmark across Vision-Language Models and GUI Grounding Datasetscited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026RESCUE: Is This Alarm Worth Acting On? Counterfactual Evaluation of VLA Runtime Monitors
- 2025Measuring Harmfulness of Computer-Using Agents
- 2026How Benchmarks Mis-Score Computer-Use Agents
- 2026From Probe Scores to Alarm Policies: Operational Validity of Activation Monitors for Language-Model Agents
- 2026Real-Time Detection and Repair of LLM Agent Failures
- 2026Runtime Skill Audit: Targeted Runtime Probing for Agent Skill Security
- 2026Runtime Compliance Verification for AI Agents
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability