Do GUI Agents Know When Not to Act? Enabling Conflict-Aware Termination for Multimodal GUI Agents
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Graphical user interface (GUI) agents are increasingly used to execute natural-language instructions on user interfaces, yet real users may issue infeasible instructions due to benign mistakes. A reliable agent should not only know how to act, but also when not to act. In this work, we introduce CONFLICTGUI, a benchmark covering instruction-internal conflicts and instruction-GUI context conflicts to study conflict-aware termination. Our evaluation reveals severe execution-biased overcompliance: agents that perform well on feasible tasks often continue to execute blindly under conflicting instructions. To mitigate this behavior, we propose CONFLICTGUARD, an inference-time framework that aligns an agent's feasibility awareness with its action generation. CONFLICTGUARD contains two coupled components: a feasibility verification protocol that guides the agent to assess instruction logic and GUI-side evidence before acting, and a conditional action modulation mechanism that steers agents from over-compliant execution into termination-oriented behavior. Experiments across five widely-used agents demonstrate that CONFLICTGUARD improves average conflict task success rate significantly, while preserving normal GUI-task performance. These results validate that a lightweight inference-time intervention can substantially boost GUI Agent's competence to identify inappropriate execution scenarios and refrain from unnecessary actions.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 20 of its 44 references that are in the map. First posted on arXiv 3 Sep 2026.
Builds on
20 of 44 references.
Closest ancestors · 6
Same direction, newest first
- 2026Faithful Mobile GUI Agents with Guided Advantage Estimator2 May
- 2026UI-Venus-1.5 Technical Report9 Feb
- 2025VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks18 Dec
- 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents9 Sep
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
Other references · 14
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
- 2023Steering Language Models With Activation Engineeringcited by 824
- 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
- 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
- 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
- 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2025The Geometry of Refusal in Large Language Models: Concept Cones and Representational Independencecited by 104
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild
- 2025XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation
- 2026Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents
- 2026Learning Reliable GUI Agents under Imperfect Priors
- 2026ActionEngine: From Reactive to Programmatic GUI Agents via State Machine Memory
- 2025Instruction Agent: Enhancing Agent with Expert Demonstration
- 2025Towards Trustworthy GUI Agents: A Survey