# Insight: Enhancing Mobile Accessibility for Blind and Visually Impaired Users with LLMs

arXiv: https://arxiv.org/abs/2605.09803 · HTML: https://mlsanity.com/paper/2605.09803/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 29 references that are in the map.
- First posted on arXiv 10 May 2026.

## Builds on (most cited first, 10 of 10)

- [A Survey on Evaluation of Large Language Models](https://mlsanity.com/paper/2307.03109/) (2023)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning](https://mlsanity.com/paper/2312.11190/) (2023)
- [GPTVoiceTasker: Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning](https://mlsanity.com/paper/2401.14268/) (2024)
- [Enabling Uniform Computer Interaction Experience for Blind Users through Large Language Models](https://mlsanity.com/paper/2407.19537/) (2024)
- [Leveraging Large Language Models for Realizing Truly Intelligent User Interfaces](https://mlsanity.com/paper/2501.12221/) (2025)

## Abstract (arXiv)

This research paper addresses the limitations of current mobile accessibility services like TalkBack, which provide manual gesture-based sequential feedback to BVI users. Motivated by the promise of large language models (LLMs), this paper introduces Insight, an Android accessibility service that provides natural language interaction and real-time summarization of the screen. The paper performs a within-subject experimental study with users to compare Insight and TalkBack on usability factors. Results show Insight reduced mental effort and task time, and was preferred because of its dialogue interface, but users felt the need for interruption management. Results show LLM-based interfaces can significantly improve mobile accessibility, and describe the potential of hybrid solutions combining gesture and dialogue modalities towards more inclusive design.