# Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents

arXiv: https://arxiv.org/abs/2605.02729 · HTML: https://mlsanity.com/paper/2605.02729/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 4 May 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 9 of 9)

- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [Model Context Protocol (MCP) at First Glance: Studying the Security and Maintainability of MCP Servers](https://mlsanity.com/paper/2506.13538/) (2025)
- [Design Principles for Generative AI Applications](https://mlsanity.com/paper/2401.14484/) (2024)
- [OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents](https://mlsanity.com/paper/2506.16042/) (2025)
- [OSExpert: Computer-Use Agents Learning Professional Skills via Exploration](https://mlsanity.com/paper/2603.07978/) (2026)
- [Establishing Heuristics for Improving the Usability of GUI Machine Learning Tools for Novice Users](https://mlsanity.com/paper/2405.08313/) (2024)

## Built on by (most cited first, 3 of 3)

- [Brick-Composer: Using MLLMs for Assembly with Diverse Bricks](https://mlsanity.com/paper/2606.05445/) (2026)
- [Affora: A Design System for Agent-Friendly Interfaces](https://mlsanity.com/paper/2609.19125/) (2026)
- [STEVE: Stabilizing Textual Gradient-Based Prompt Optimization via Error-Driven Refinement and Regularized Verification](https://mlsanity.com/paper/2609.23716/) (2026)

## Abstract (arXiv)

Recent advances have enabled general computer-use agents that interpret screens and execute grounded actions from human instructions, yet they still struggle to generalize to unseen and evolving interfaces. While improving agent capability remains important, agent compatible interface design offers a complementary path by aligning interaction semantics with agent prior knowledge. In this paper, we revisit Nielsen 10 usability heuristics through the lens of computer-use agents, identifying which principles naturally transfer, where implicit design assumptions create agent specific failures, and how safe additive augmentations can improve robustness without harming human usability. To evaluate these ideas, we introduce UI-Verse, a suite of controlled environments built around functionally similar interfaces with different applied heuristics. Experiments show that our augmented heuristics consistently improve task completion and modestly improve efficiency, with combined heuristics yielding further gains. Human studies further show that these designs preserve the original interaction workflow without observable usability regressions. Overall, our findings highlight interface design as a practical complementary avenue for improving the reliability and generalization of computer use agents.