# GamerAstra: Supporting 2D Non-Twitch Video Games for Blind and Low-Vision Players through a Multi-Agent Framework

arXiv: https://arxiv.org/abs/2506.22937 · HTML: https://mlsanity.com/paper/2506.22937/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 22 of its 65 references that are in the map.
- First posted on arXiv 28 Jun 2025.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 22 of 22)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [PP-OCR: A Practical Ultra Lightweight OCR System](https://mlsanity.com/paper/2009.09941/) (2020)
- [Cradle: Empowering Foundation Agents Towards General Computer Control](https://mlsanity.com/paper/2403.03186/) (2024)
- [A Survey on Large Language Model-Based Game Agents](https://mlsanity.com/paper/2404.02039/) (2024)
- [LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editing](https://mlsanity.com/paper/2402.10294/) (2024)
- [AppAgentX: Evolving GUI Agents as Proficient Smartphone Users](https://mlsanity.com/paper/2503.02268/) (2025)
- [EmoVoice: LLM-based Emotional Text-To-Speech Model with Freestyle Text Prompting](https://mlsanity.com/paper/2504.12867/) (2025)
- [VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planning](https://mlsanity.com/paper/2312.11190/) (2023)
- [WorldScribe: Towards Context-Aware Live Visual Descriptions](https://mlsanity.com/paper/2408.06627/) (2024)
- [EmoSphere-TTS: Emotional Style and Intensity Modeling via Spherical Emotion Vector for Controllable Emotional Text-to-Speech](https://mlsanity.com/paper/2406.07803/) (2024)
- [Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case](https://mlsanity.com/paper/2409.12889/) (2024)
- [Making Short-Form Videos Accessible with Hierarchical Video Summaries](https://mlsanity.com/paper/2402.10382/) (2024)
- [VizAbility: Enhancing Chart Accessibility with LLM-based Conversational Interaction](https://mlsanity.com/paper/2310.09611/) (2023)
- [Will GPT-4 Run DOOM?](https://mlsanity.com/paper/2403.05468/) (2024)
- [DanmuA11y: Making Time-Synced On-Screen Video Comments (Danmu) Accessible to Blind and Low Vision Users via Multi-Viewer Audio Discussions](https://mlsanity.com/paper/2501.15711/) (2025)
- [NavStick: Making Video Games Blind-Accessible via the Ability to Look Around](https://mlsanity.com/paper/2109.01202/) (2021)
- ["The Guide Has Your Back": Exploring How Sighted Guides Can Enhance Accessibility in Social Virtual Reality for Blind and Low Vision People](https://mlsanity.com/paper/2410.21659/) (2024)
- [How Users Who are Blind or Low Vision Play Mobile Games: Perceptions, Challenges, and Strategies](https://mlsanity.com/paper/2502.09866/) (2025)
- [DesignChecker: Visual Design Support for Blind and Low Vision Web Developers](https://mlsanity.com/paper/2407.17681/) (2024)

## Abstract (arXiv)

Blind and low-vision (BLV) players face critical challenges in engaging with video games due to the inaccessibility of visual elements, difficulties navigating interfaces, and limitations in performing interaction. Meanwhile, the development of specialized accessibility features typically requires substantial programming effort and is often implemented on a game-by-game basis. To address these challenges, we introduce GamerAstra, a multi-agent human-AI collaboration framework that leverages a multi-agent design to facilitate access to 2D non-twitch video games for BLV players. It integrates vision-language models and computer vision techniques, enabling interaction with games lacking native accessibility support. The framework also incorporates custom assistance granularities to support varying degrees of visual impairment and enhances interface navigation through multiple input modalities. Technical evaluations and user studies indicate that GamerAstra effectively enhances playability and provides a more immersive gaming experience for BLV players. These findings also underscore potential avenues for advancing intelligent accessibility frameworks in the gaming domain.