# A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents

arXiv: https://arxiv.org/abs/2602.09310 · HTML: https://mlsanity.com/paper/2602.09310/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 74 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 25 of 25)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [GAIA: a benchmark for General AI Assistants](https://mlsanity.com/paper/2311.12983/) (2023)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Social Simulacra: Creating Populated Prototypes for Social Computing Systems](https://mlsanity.com/paper/2208.04024/) (2022)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [LLM Agents Grounded in Self-Reports Enable General-Purpose Simulation of Individuals](https://mlsanity.com/paper/2411.10109/) (2024)
- [WebLINX: Real-World Website Navigation with Multi-Turn Dialogue](https://mlsanity.com/paper/2402.05930/) (2024)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [UFO: A UI-Focused Agent for Windows OS Interaction](https://mlsanity.com/paper/2402.07939/) (2024)
- [ScreenAgent: A Vision Language Model-driven Computer Control Agent](https://mlsanity.com/paper/2402.07945/) (2024)
- [WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences](https://mlsanity.com/paper/2306.07906/) (2023)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [AgentStudio: A Toolkit for Building General Virtual Agents](https://mlsanity.com/paper/2403.17918/) (2024)
- [GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding](https://mlsanity.com/paper/2406.10819/) (2024)
- [CowPilot: A Framework for Autonomous and Human-Agent Collaborative Web Navigation](https://mlsanity.com/paper/2501.16609/) (2025)
- [AXNav: Replaying Accessibility Tests from Natural Language](https://mlsanity.com/paper/2310.02424/) (2023)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models](https://mlsanity.com/paper/2407.19537/) (2024)
- [Finger Based Technique (FBT): An Innovative System for Improved Usability for the Blind Users' Dynamic Interaction with Mobile Touch Screen Devices](https://mlsanity.com/paper/1708.05073/) (2017)

## Built on by (most cited first, 3 of 3)

- ["Zooming In" on Agentic Web Browsers as Assistive Technologies: A Case Study with a Low-Vision Technology Expert](https://mlsanity.com/paper/2606.24870/) (2026)
- [Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents](https://mlsanity.com/paper/2607.17527/) (2026)
- [Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications](https://mlsanity.com/paper/2609.00524/) (2026)

## Abstract (arXiv)

Computer Use Agents (CUAs) operate interfaces by pointing, clicking, and typing -- mirroring interactions of sighted users (SUs) who can thus monitor CUAs and share control. CUAs do not reflect interactions by blind and low-vision users (BLVUs) who use assistive technology (AT). BLVUs thus cannot easily collaborate with CUAs. To characterize the accessibility gap of CUAs, we present A11y-CUA, a dataset of BLVUs and SUs performing 60 everyday tasks with 40.4 hours and 158,325 events. Our dataset analysis reveals that our collected interaction traces quantitatively confirm distinct interaction styles between SU and BLVU groups (mouse- vs. keyboard-dominant) and demonstrate interaction diversity within each group (sequential vs. shortcut navigation for BLVUs). We then compare collected traces to state-of-the-art CUAs under default and AT conditions (keyboard-only, magnifier). The default CUA executed 78.3% of tasks successfully. But with the AT conditions, CUA's performance dropped to 41.67% and 28.3% with keyboard-only and magnifier conditions respectively, and did not reflect nuances of real AT use. With our open A11y-CUA dataset, we aim to promote collaborative and accessible CUAs for everyone.