# MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs

arXiv: https://arxiv.org/abs/2403.13690 · HTML: https://mlsanity.com/paper/2403.13690/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 17 of its 81 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 17 of 17)

- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)
- [Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning](https://mlsanity.com/paper/2003.00380/) (2020)
- [Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning](https://mlsanity.com/paper/1902.11247/) (2019)
- [Owl Eyes: Spotting UI Display Issues via Visual Understanding](https://mlsanity.com/paper/2009.01417/) (2020)
- [Automated Reporting of GUI Design Violations for Mobile Apps](https://mlsanity.com/paper/1802.04732/) (2018)
- [Translating Video Recordings of Mobile App Usages into Replayable Scenarios](https://mlsanity.com/paper/2005.09057/) (2020)
- [AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces](https://mlsanity.com/paper/2303.06782/) (2023)
- [Continuous, Evolutionary and Large-Scale: A New Perspective for Automated Mobile App Testing](https://mlsanity.com/paper/1801.06267/) (2018)
- [How do Developers Test Android Applications?](https://mlsanity.com/paper/1801.06268/) (2018)
- [Detecting and Summarizing GUI Changes in Evolving Mobile Apps](https://mlsanity.com/paper/1807.09440/) (2018)
- [Mining Android App Usages for Generating Actionable GUI-based Execution Scenarios](https://mlsanity.com/paper/1801.06271/) (2018)
- [VET: Identifying and Avoiding UI Exploration Tarpits](https://mlsanity.com/paper/2102.06377/) (2021)
- [V2S: A Tool for Translating Video Recordings of Mobile App Usages into Replayable Scenarios](https://mlsanity.com/paper/2103.04531/) (2021)
- [Translating Video Recordings of Complex Mobile App UI Gestures into Replayable Scenarios](https://mlsanity.com/paper/2301.01191/) (2023)
- [Optimization of Switch Keyboards](https://mlsanity.com/paper/1503.04375/) (2015)
- [TangibleGrid: Tangible Web Layout Design for Blind Users](https://mlsanity.com/paper/2208.08482/) (2022)

## Abstract (arXiv)

Recent research has begun to examine the potential of automatically finding and fixing accessibility issues that manifest in software. However, while recent work makes important progress, it has generally been skewed toward identifying issues that affect users with certain disabilities, such as those with visual or hearing impairments. However, there are other groups of users with different types of disabilities that also need software tooling support to improve their experience. As such, this paper aims to automatically identify accessibility issues that affect users with motor-impairments. To move toward this goal, this paper introduces a novel approach, called MotorEase, capable of identifying accessibility issues in mobile app UIs that impact motor-impaired users. Motor-impaired users often have limited ability to interact with touch-based devices, and instead may make use of a switch or other assistive mechanism -- hence UIs must be designed to support both limited touch gestures and the use of assistive devices. MotorEase adapts computer vision and text processing techniques to enable a semantic understanding of app UI screens, enabling the detection of violations related to four popular, previously unexplored UI design guidelines that support motor-impaired users, including: (i) visual touch target size, (ii) expanding sections, (iii) persisting elements, and (iv) adjacent icon visual distance. We evaluate MotorEase on a newly derived benchmark, called MotorCheck, that contains 555 manually annotated examples of violations to the above accessibility guidelines, across 1599 screens collected from 70 applications via a mobile app testing tool. Our experiments illustrate that MotorEase is able to identify violations with an average accuracy of ~90%, and a false positive rate of less than 9%, outperforming baseline techniques.