# Computational Approaches for App-to-App Retrieval and Design Consistency Check

arXiv: https://arxiv.org/abs/2309.10328 · HTML: https://mlsanity.com/paper/2309.10328/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 33 references that are in the map.
- First posted on arXiv 19 Sep 2023.

## Builds on (most cited first, 9 of 9)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [VINS: Visual Search for Mobile User Interface Design](https://mlsanity.com/paper/2102.05216/) (2021)
- [Wireframe-Based UI Design Search Through Image Autoencoder](https://mlsanity.com/paper/2103.07085/) (2021)

## Built on by (most cited first, 3 of 3)

- [Assessing Graphical Perception of Image Embedding Models using Channel Effectiveness](https://mlsanity.com/paper/2407.20845/) (2024)
- [Bridging Gulfs in UI Generation through Semantic Guidance](https://mlsanity.com/paper/2601.19171/) (2026)
- [GhostUI: Unveiling Hidden Interactions in Mobile UI](https://mlsanity.com/paper/2601.19258/) (2026)

## Abstract (arXiv)

Extracting semantic representations from mobile user interfaces (UI) and using the representations for designers' decision-making processes have shown the potential to be effective computational design support tools. Current approaches rely on machine learning models trained on small-sized mobile UI datasets to extract semantic vectors and use screenshot-to-screenshot comparison to retrieve similar-looking UIs given query screenshots. However, the usability of these methods is limited because they are often not open-sourced and have complex training pipelines for practitioners to follow, and are unable to perform screenshot set-to-set (i.e., app-to-app) retrieval. To this end, we (1) employ visual models trained with large web-scale images and test whether they could extract a UI representation in a zero-shot way and outperform existing specialized models, and (2) use mathematically founded methods to enable app-to-app retrieval and design consistency analysis. Our experiments show that our methods not only improve upon previous retrieval models but also enable multiple new applications.