# When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agents

arXiv: https://arxiv.org/abs/2602.08995 · HTML: https://mlsanity.com/paper/2602.08995/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 37 of its 53 references that are in the map.
- First posted on arXiv 9 Feb 2026.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)
- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 30 of 37)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://mlsanity.com/paper/2302.12173/) (2023)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [Alignment faking in large language models](https://mlsanity.com/paper/2412.14093/) (2024)
- [R-Judge: Benchmarking Safety Risk Awareness for LLM Agents](https://mlsanity.com/paper/2401.10019/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [An Illusion of Progress? Assessing the Current State of Web Agents](https://mlsanity.com/paper/2504.01382/) (2025)
- [OpenCUA: Open Foundations for Computer-Use Agents](https://mlsanity.com/paper/2508.09123/) (2025)
- [Agentic Misalignment: How LLMs Could Be Insider Threats](https://mlsanity.com/paper/2510.05179/) (2025)
- [WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks](https://mlsanity.com/paper/2504.18575/) (2025)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning](https://mlsanity.com/paper/2406.09187/) (2024)
- [OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents](https://mlsanity.com/paper/2506.14866/) (2025)
- [PromptArmor: Simple yet Effective Prompt Injection Defenses](https://mlsanity.com/paper/2507.15219/) (2025)
- [The Task Shield: Enforcing Task Alignment to Defend Against Indirect Prompt Injection in LLM Agents](https://mlsanity.com/paper/2412.16682/) (2024)
- [AgentAuditor: Human-Level Safety and Security Evaluation for LLM Agents](https://mlsanity.com/paper/2506.00641/) (2025)
- [AGrail: A Lifelong Agent Guardrail with Effective and Adaptive Safety Detection](https://mlsanity.com/paper/2502.11448/) (2025)
- [RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environments](https://mlsanity.com/paper/2505.21936/) (2025)
- [RiOSWorld: Benchmarking the Risk of Multimodal Computer-Use Agents](https://mlsanity.com/paper/2506.00618/) (2025)
- [DoomArena: A framework for Testing AI Agents Against Evolving Security Threats](https://mlsanity.com/paper/2504.14064/) (2025)
- [Building a Foundational Guardrail for General Agentic Systems via Synthetic Data](https://mlsanity.com/paper/2510.09781/) (2025)
- [AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agents](https://mlsanity.com/paper/2509.07764/) (2025)
- [A Survey on the Safety and Security Threats of Computer-Using Agents: JARVIS or Ultron?](https://mlsanity.com/paper/2505.10924/) (2025)
- [Testing and Understanding Erroneous Planning in LLM Agents through Synthesized User Inputs](https://mlsanity.com/paper/2404.17833/) (2024)

## Built on by (most cited first, 8 of 8)

- [SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction](https://mlsanity.com/paper/2606.02540/) (2026)
- [Safeguarding LLM Agents from Misalignment through Provenance Analysis](https://mlsanity.com/paper/2607.01236/) (2026)
- [ROGUE: Evaluating Corrigibility Failures in Frontier Computer-Use Agents](https://mlsanity.com/paper/2606.00341/) (2026)
- [Triaging Threats to Specialized Guardrails](https://mlsanity.com/paper/2605.30693/) (2026)
- [A History-Aware Visually Grounded Critic for Computer Use Agents](https://mlsanity.com/paper/2606.11078/) (2026)
- [Don't Offer What Can't Be Done: Deterministic Executability Gating for LLM Skill Selection at Scale](https://mlsanity.com/paper/2608.01050/) (2026)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)
- [OSGuard: A Benchmark for Safety in Computer-Use Agents](https://mlsanity.com/paper/2606.15034/) (2026)

## Abstract (arXiv)

Computer-use agents (CUAs) have made tremendous progress in the past year, yet they still frequently produce misaligned actions that deviate from the user's original intent. Such misaligned actions may arise from external attacks (e.g., indirect prompt injection) or from internal limitations (e.g., erroneous reasoning). They not only expose CUAs to safety risks, but also degrade task efficiency and reliability. This work makes the first effort to define and study misaligned action detection in CUAs, with comprehensive coverage of both externally induced and internally arising misaligned actions. We further identify three common categories in real-world CUA deployment and construct MisActBench, a benchmark of realistic trajectories with human-annotated, action-level alignment labels. Moreover, we propose DeAction, a practical and universal guardrail that detects misaligned actions before execution and iteratively corrects them through structured feedback. DeAction outperforms all existing baselines across offline and online evaluations with moderate latency overhead: (1) On MisActBench, it outperforms baselines by over 15% absolute in F1 score; (2) In online evaluation, it reduces attack success rate by over 90% under adversarial settings while preserving or even improving task success rate in benign environments.