mlsanityData to 7 Oct 2026

Are We There Yet? Assessing Computer-Use Agents for Blind Users' Accessible Interaction with Desktop Applications

arXiv 2609.00524 ↗First posted 1 Sep 2026cs.HC cs.AIcited by 0

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Computer-use agents are emerging as a paradigm for agentic human-AI interaction, combining language reasoning with multi-modal interface grounding to operate GUIs. Yet their effectiveness for blind screen-reader users in real-world desktop workflows remains unclear. We present a three-week diary study with 8 blind users using OLLA, a screen-reader-accessible CUA prototype, collecting 1,258 commands across 12 applications with screenshots, UI trees, model responses, and action traces. We evaluate GPT-5 during deployment and re-execute the same commands with four additional models. GPT-5 achieved the highest success rate at 52.5%. Trace analysis reveals grounding, planning, constraint-tracking, and termination failures, while interviews reveal beyond-automation needs.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 23 of its 69 references that are in the map. First posted on arXiv 1 Sep 2026.

Builds on

23 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2026A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents10 Feb
  2. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
  3. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  6. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
  7. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Taskscited by 306
  8. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  9. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  10. 2024ST-WebAgentBench: A Benchmark for Evaluating Safety and Trustworthiness in Web Agentscited by 126
  11. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Modelscited by 98
  12. 2023"No, to the Right" -- Online Language Corrections for Robotic Manipulation via Shared Autonomycited by 60
8 more
  1. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  2. 2025Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directednesscited by 8
  3. 2025OKBench: Democratizing LLM Evaluation with Fully Automated, On-Demand, Open Knowledge Benchmarkingcited by 8
  4. 2026Tracking the Limits of Knowledge Propagation: How LLMs Fail at Multi-Step Reasoning with Conflicting Knowledgecited by 7
  5. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?cited by 6
  6. 2026From Struggle to Success: Context-Aware Guidance for Screen Reader Users in Computer Usecited by 2
  7. 2026Finding the Signal in the Noise: An Exploratory Study on Assessing the Effectiveness of AI and Accessibility Forums for Blind Users' Support Needscited by 1
  8. 2026Capability Self-Assessment in Large Language Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
  2. 2026Position: Assistive Agents Need Accessibility Alignment
  3. 2026Explainable AI for Blind and Low-Vision Users: Navigating Trust, Modality, and Interpretability in the Agentic Era
  4. 2026Affora: A Design System for Agent-Friendly Interfaces
  5. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  6. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction
  7. 2025Computer-Use Agents as Judges for Generative User Interface
  8. 2026Agent-Computer Observation Interfaces Enable Dynamic Computer Use