# AXNav: Replaying Accessibility Tests from Natural Language

arXiv: https://arxiv.org/abs/2310.02424 · HTML: https://mlsanity.com/paper/2310.02424/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 68 references that are in the map.
- First posted on arXiv 3 Oct 2023.

## Builds on (most cited first, 24 of 24)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Gorilla: Large Language Model Connected with Massive APIs](https://mlsanity.com/paper/2305.15334/) (2023)
- [A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis](https://mlsanity.com/paper/2307.12856/) (2023)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Understanding the Role of Human Intuition on Reliance in Human-AI Decision-Making with Explanations](https://mlsanity.com/paper/2301.07255/) (2023)
- [From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfaces](https://mlsanity.com/paper/2306.00245/) (2023)
- [Human-AI Collaboration via Conditional Delegation: A Case Study of Content Moderation](https://mlsanity.com/paper/2204.11788/) (2022)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)
- [Prompting Is All You Need: Automated Android Bug Replay with Large Language Models](https://mlsanity.com/paper/2306.01987/) (2023)
- [UGIF: UI Grounded Instruction Following](https://mlsanity.com/paper/2211.07615/) (2022)
- [Chatting with GPT-3 for Zero-Shot Human-Like Mobile Automated GUI Testing](https://mlsanity.com/paper/2305.09434/) (2023)
- [Responsible Task Automation: Empowering Large Language Models as Responsible Task Automators](https://mlsanity.com/paper/2306.01242/) (2023)
- [Accessibility in Software Practice: A Practitioner's Perspective](https://mlsanity.com/paper/2103.08778/) (2021)
- [How do Developers Test Android Applications?](https://mlsanity.com/paper/1801.06268/) (2018)
- [Accessible or Not? An Empirical Investigation of Android App Accessibility](https://mlsanity.com/paper/2203.06422/) (2022)
- [To the Attention of Mobile Software Developers: Guess What, Test your App!](https://mlsanity.com/paper/1902.02610/) (2019)

## Built on by (most cited first, 30 of 31)

- [Large Language Model-Based Agents for Software Engineering: A Survey](https://mlsanity.com/paper/2409.02977/) (2024)
- [LLM-Based Multi-Agent Systems for Software Engineering: Literature Review, Vision and the Road Ahead](https://mlsanity.com/paper/2404.04834/) (2024)
- [UXAgent: An LLM Agent-Based Usability Testing Framework for Web Design](https://mlsanity.com/paper/2502.12561/) (2025)
- [Repo2Run: Automated Building Executable Environment for Code Repository at Scale](https://mlsanity.com/paper/2502.13681/) (2025)
- [LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automation](https://mlsanity.com/paper/2404.16054/) (2024)
- [LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospects](https://mlsanity.com/paper/2504.19838/) (2025)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [LLM Agent Meets Agentic AI: Can LLM Agents Simulate Customers to Evaluate Agentic-AI-based Shopping Assistants?](https://mlsanity.com/paper/2509.21501/) (2025)
- [From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impacts](https://mlsanity.com/paper/2410.09006/) (2024)
- [UXAgent: A System for Simulating Usability Testing of Web Design with LLM Agents](https://mlsanity.com/paper/2504.09407/) (2025)
- [AutoEval: A Practical Framework for Autonomous Evaluation of Mobile Agents](https://mlsanity.com/paper/2503.02403/) (2025)
- [EcoAgent: An Efficient Device-Cloud Collaborative Multi-Agent Framework for Mobile Automation](https://mlsanity.com/paper/2505.05440/) (2025)
- [SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agents](https://mlsanity.com/paper/2510.11035/) (2025)
- [Mapping the Design Space of User Experience for Computer Use Agents](https://mlsanity.com/paper/2602.07283/) (2026)
- [CodeVisionary: An Agent-based Framework for Evaluating Large Language Models in Code Generation](https://mlsanity.com/paper/2504.13472/) (2025)
- [OpsAgent: An Evolving Multi-agent System for Incident Management in Microservices](https://mlsanity.com/paper/2510.24145/) (2025)
- [UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity](https://mlsanity.com/paper/2409.04081/) (2024)
- [WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements](https://mlsanity.com/paper/2602.11724/) (2026)
- [Designing LLM-based Multi-Agent Systems for Software Engineering Tasks: Quality Attributes, Design Patterns and Rationale](https://mlsanity.com/paper/2511.08475/) (2025)
- [A11y-CUA Dataset: Characterizing the Accessibility Gap in Computer Use Agents](https://mlsanity.com/paper/2602.09310/) (2026)
- [Generative transformations and patterns in LLM-native approaches for software verification and falsification](https://mlsanity.com/paper/2404.09384/) (2024)
- [ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models](https://mlsanity.com/paper/2504.02110/) (2025)
- [PaperRepro: Automated Computational Reproducibility Assessment for Social Science Papers](https://mlsanity.com/paper/2603.00058/) (2026)
- [GUI Test Migration via Abstraction and Concretization](https://mlsanity.com/paper/2409.05028/) (2024)
- [AgentLens: Adaptive Visual Modalities for Human-Agent Interaction in Mobile GUI Agents](https://mlsanity.com/paper/2604.20279/) (2026)
- [UXBench: Measuring the Actionability of LLM-Generated UX Critiques](https://mlsanity.com/paper/2606.16262/) (2026)
- [Not There Yet: Evaluating Vision Language Models in Simulating the Visual Perception of People with Low Vision](https://mlsanity.com/paper/2508.10972/) (2025)
- [From User Interface to Agent Interface: Efficiency Optimization of UI Representations for LLM Agents](https://mlsanity.com/paper/2512.13438/) (2025)
- [Sidekick: Designing Communication for Effective Multitasking with Computer Use Agents](https://mlsanity.com/paper/2607.17527/) (2026)

## Abstract (arXiv)

Developers and quality assurance testers often rely on manual testing to test accessibility features throughout the product lifecycle. Unfortunately, manual testing can be tedious, often has an overwhelming scope, and can be difficult to schedule amongst other development milestones. Recently, Large Language Models (LLMs) have been used for a variety of tasks including automation of UIs, however to our knowledge no one has yet explored their use in controlling assistive technologies for the purposes of supporting accessibility testing. In this paper, we explore the requirements of a natural language based accessibility testing workflow, starting with a formative study. From this we build a system that takes as input a manual accessibility test (e.g., ``Search for a show in VoiceOver'') and uses an LLM combined with pixel-based UI Understanding models to execute the test and produce a chaptered, navigable video. In each video, to help QA testers we apply heuristics to detect and flag accessibility issues (e.g., Text size not increasing with Large Text enabled, VoiceOver navigation loops). We evaluate this system through a 10 participant user study with accessibility QA professionals who indicated that the tool would be very useful in their current work and performed tests similarly to how they would manually test the features. The study also reveals insights for future work on using LLMs for accessibility testing.