# Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models

arXiv: https://arxiv.org/abs/2407.19537 · HTML: https://mlsanity.com/paper/2407.19537/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Jul 2024.

## Built on by (most cited first, 9 of 9)

- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Surfacing Variations to Calibrate Perceived Reliability of MLLM-generated Image Descriptions](https://mlsanity.com/paper/2507.15692/) (2025)
- [Permission Manifests for Web Agents](https://mlsanity.com/paper/2601.02371/) (2025)
- [AccessGuru: Leveraging LLMs to Detect and Correct Web Accessibility Violations in HTML Code](https://mlsanity.com/paper/2507.19549/) (2025)
- [A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents](https://mlsanity.com/paper/2602.09310/) (2026)
- [Finding the Signal in the Noise: An Exploratory Study on Assessing the Effectiveness of AI and Accessibility Forums for Blind Users' Support Needs](https://mlsanity.com/paper/2602.18623/) (2026)
- [Lost in Instructions: Study of Blind Users' Experiences with DIY Manuals and AI-Rewritten Instructions for Assembly, Operation, and Troubleshooting of Tangible Products](https://mlsanity.com/paper/2602.18630/) (2026)
- [Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs](https://mlsanity.com/paper/2605.09803/) (2026)
- [From Plots to Words: Model-Aware Multimodal Explanations as a Foundation for Accessible, Non-Visual Interaction](https://mlsanity.com/paper/2608.24910/) (2026)

## Abstract (arXiv)

Blind individuals, who by necessity depend on screen readers to interact with computers, face considerable challenges in navigating the diverse and complex graphical user interfaces of different computer applications. The heterogeneity of various application interfaces often requires blind users to remember different keyboard combinations and navigation methods to use each application effectively. To alleviate this significant interaction burden imposed by heterogeneous application interfaces, we present Savant, a novel assistive technology powered by large language models (LLMs) that allows blind screen reader users to interact uniformly with any application interface through natural language. Novelly, Savant can automate a series of tedious screen reader actions on the control elements of the application when prompted by a natural language command from the user. These commands can be flexible in the sense that the user is not strictly required to specify the exact names of the control elements in the command. A user study evaluation of Savant with 11 blind participants demonstrated significant improvements in interaction efficiency and usability compared to current practices.