# GreaseVision: Rewriting the Rules of the Interface

arXiv: https://arxiv.org/abs/2204.03731 · HTML: https://mlsanity.com/paper/2204.03731/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 98 references that are in the map.
- First posted on arXiv 7 Apr 2022.

## Builds on (most cited first, 25 of 25)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [What is being transferred in transfer learning?](https://mlsanity.com/paper/2008.11687/) (2020)
- [Rapid Learning or Feature Reuse? Towards Understanding the Effectiveness of MAML](https://mlsanity.com/paper/1909.09157/) (2019)
- [EAST: An Efficient and Accurate Scene Text Detector](https://mlsanity.com/paper/1704.03155/) (2017)
- [Learning from the Worst: Dynamically Generated Datasets to Improve Online Hate Detection](https://mlsanity.com/paper/2012.15761/) (2020)
- [A Survey of Human-in-the-loop for Machine Learning](https://mlsanity.com/paper/2108.00941/) (2021)
- [pix2code: Generating Code from a Graphical User Interface Screenshot](https://mlsanity.com/paper/1705.07962/) (2017)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [On the Role of Neural Collapse in Transfer Learning](https://mlsanity.com/paper/2112.15121/) (2021)
- [ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces](https://mlsanity.com/paper/2012.12350/) (2020)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Screen Parsing: Towards Reverse Engineering of UI Models from Screenshots](https://mlsanity.com/paper/2109.08763/) (2021)
- ['I Just Want to Hack Myself to Not Get Distracted': Evaluating Design Interventions for Self-Control on Facebook](https://mlsanity.com/paper/2001.04180/) (2020)
- [GUIGAN: Learning to Generate GUI Designs Using Generative Adversarial Networks](https://mlsanity.com/paper/2101.09978/) (2021)
- [GenNI: Human-AI Collaboration for Data-Backed Text Generation](https://mlsanity.com/paper/2110.10185/) (2021)
- [Optimizing User Interface Layouts via Gradient Descent](https://mlsanity.com/paper/2002.10702/) (2020)
- [Are iPhones Really Better for Privacy? Comparative Study of iOS and Android Apps](https://mlsanity.com/paper/2109.13722/) (2021)
- [Third Party Tracking in the Mobile Ecosystem](https://mlsanity.com/paper/1804.03603/) (2018)
- [I Want My App That Way: Reclaiming Sovereignty Over Personal Devices](https://mlsanity.com/paper/2102.11819/) (2021)
- [Low-Loss Subspace Compression for Clean Gains against Multi-Agent Backdoor Attacks](https://mlsanity.com/paper/2203.03692/) (2022)
- [Characteristics and Challenges of Low-Code Development: The Practitioners' Perspective](https://mlsanity.com/paper/2107.07482/) (2021)
- [Mind-proofing Your Phone: Navigating the Digital Minefield with GreaseTerminator](https://mlsanity.com/paper/2112.10699/) (2021)

## Built on by (most cited first, 2 of 2)

- [Cross-Reality Re-Rendering: Manipulating between Digital and Physical Realities](https://mlsanity.com/paper/2211.08005/) (2022)
- [From Exploration to Revelation: Detecting Dark Patterns in Mobile Apps](https://mlsanity.com/paper/2411.18084/) (2024)

## Abstract (arXiv)

Digital harms can manifest across any interface. Key problems in addressing these harms include the high individuality of harms and the fast-changing nature of digital systems. As a result, we still lack a systematic approach to study harms and produce interventions for end-users. We put forward GreaseVision, a new framework that enables end-users to collaboratively develop interventions against harms in software using a no-code approach and recent advances in few-shot machine learning. The contribution of the framework and tool allow individual end-users to study their usage history and create personalized interventions. Our contribution also enables researchers to study the distribution of harms and interventions at scale.