# AidUI: Toward Automated Recognition of Dark Patterns in User Interfaces

arXiv: https://arxiv.org/abs/2303.06782 · HTML: https://mlsanity.com/paper/2303.06782/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 56 references that are in the map.
- First posted on arXiv 12 Mar 2023.

## Builds on (most cited first, 13 of 13)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixels](https://mlsanity.com/paper/2101.04893/) (2021)
- [Machine Learning-Based Prototyping of Graphical User Interfaces for Mobile Apps](https://mlsanity.com/paper/1802.02312/) (2018)
- [Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots](https://mlsanity.com/paper/2109.08763/) (2021)
- [Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?](https://mlsanity.com/paper/2008.05132/) (2020)
- [Unblind Your Apps: Predicting Natural-Language Labels for Mobile GUI Components by Deep Learning](https://mlsanity.com/paper/2003.00380/) (2020)
- [(Un)informed Consent: Studying GDPR Consent Notices in the Field](https://mlsanity.com/paper/1909.02638/) (2019)
- [Owl Eyes: Spotting UI Display Issues via Visual Understanding](https://mlsanity.com/paper/2009.01417/) (2020)
- [Automated Reporting of GUI Design Violations for Mobile Apps](https://mlsanity.com/paper/1802.04732/) (2018)
- [Translating Video Recordings of Mobile App Usages into Replayable Scenarios](https://mlsanity.com/paper/2005.09057/) (2020)
- [MadDroid: Characterising and Detecting Devious Ad Content for Android Apps](https://mlsanity.com/paper/2002.01656/) (2020)

## Built on by (most cited first, 9 of 9)

- [Investigating the Impact of Dark Patterns on LLM-Based Web Agents](https://mlsanity.com/paper/2510.18113/) (2025)
- [Automatically Detecting Online Deceptive Patterns](https://mlsanity.com/paper/2411.07441/) (2024)
- [Why is the User Interface a Dark Pattern? : Explainable Auto-Detection and its Analysis](https://mlsanity.com/paper/2401.04119/) (2023)
- [MotorEase: Automated Detection of Motor Impairment Accessibility Issues in Mobile App UIs](https://mlsanity.com/paper/2403.13690/) (2024)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)
- [Detecting Dark Patterns in User Interfaces Using Logistic Regression and Bag-of-Words Representation](https://mlsanity.com/paper/2412.14187/) (2024)
- [On the Suitability of LLM-Driven Agents for Dark Pattern Audits](https://mlsanity.com/paper/2603.03881/) (2026)
- [Learning to Detect UI Principle Violations via Reinforcement Learning](https://mlsanity.com/paper/2607.20690/) (2026)
- [AI Watchdog: Agent Interfaces for Detecting and Defending Against Manipulative Dark Patterns in AI Conversations](https://mlsanity.com/paper/2608.21841/) (2026)

## Abstract (arXiv)

Past studies have illustrated the prevalence of UI dark patterns, or user interfaces that can lead end-users toward (unknowingly) taking actions that they may not have intended. Such deceptive UI designs can result in adverse effects on end users, such as oversharing personal information or financial loss. While significant research progress has been made toward the development of dark pattern taxonomies, developers and users currently lack guidance to help recognize, avoid, and navigate these often subtle design motifs. However, automated recognition of dark patterns is a challenging task, as the instantiation of a single type of pattern can take many forms, leading to significant variability. In this paper, we take the first step toward understanding the extent to which common UI dark patterns can be automatically recognized in modern software applications. To do this, we introduce AidUI, a novel automated approach that uses computer vision and natural language processing techniques to recognize a set of visual and textual cues in application screenshots that signify the presence of ten unique UI dark patterns, allowing for their detection, classification, and localization. To evaluate our approach, we have constructed ContextDP, the current largest dataset of fully-localized UI dark patterns that spans 175 mobile and 83 web UI screenshots containing 301 dark pattern instances. The results of our evaluation illustrate that AidUI achieves an overall precision of 0.66, recall of 0.67, F1-score of 0.65 in detecting dark pattern instances, reports few false positives, and is able to localize detected patterns with an IoU score of ~0.84. Furthermore, a significant subset of our studied dark patterns can be detected quite reliably (F1 score of over 0.82), and future research directions may allow for improved detection of additional patterns.