# Automatic Macro Mining from Interaction Traces at Scale

arXiv: https://arxiv.org/abs/2310.07023 · HTML: https://mlsanity.com/paper/2310.07023/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 39 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM 2 Technical Report](https://mlsanity.com/paper/2305.10403/) (2023)
- [Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models](https://mlsanity.com/paper/2108.08877/) (2021)
- [Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understanding](https://mlsanity.com/paper/2210.03347/) (2022)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [Mapping Natural Language Instructions to Mobile UI Action Sequences](https://mlsanity.com/paper/2005.03776/) (2020)
- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020)
- [UIBert: Learning Generic Multimodal Representations for UI Understanding](https://mlsanity.com/paper/2107.13731/) (2021)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [Spotlight: Mobile UI Understanding using Vision-Language Models with a Focus](https://mlsanity.com/paper/2209.14927/) (2022)
- [A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility](https://mlsanity.com/paper/2202.02312/) (2022)
- [ActionBert: Leveraging User Actions for Semantic Understanding of User Interfaces](https://mlsanity.com/paper/2012.12350/) (2020)
- [Screen2Vec: Semantic Embedding of GUI Screens and GUI Components](https://mlsanity.com/paper/2101.11103/) (2021)
- [Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scale](https://mlsanity.com/paper/2201.04100/) (2022)
- [Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations](https://mlsanity.com/paper/1909.00031/) (2019)
- [Towards Better Semantic Understanding of Mobile Interfaces](https://mlsanity.com/paper/2210.02663/) (2022)
- [InstructableCrowd: Creating IF-THEN Rules for Smartphones via Conversations with the Crowd](https://mlsanity.com/paper/1909.05725/) (2019)

## Built on by (most cited first, 12 of 12)

- [WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?](https://mlsanity.com/paper/2403.07718/) (2024)
- [Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automation](https://mlsanity.com/paper/2312.03003/) (2023)
- [Morae: Proactively Pausing UI Agents for User Choices](https://mlsanity.com/paper/2508.21456/) (2025)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [LLM-Explorer: Towards Efficient and Affordable LLM-based Exploration for Mobile Apps](https://mlsanity.com/paper/2505.10593/) (2025)
- [UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity](https://mlsanity.com/paper/2409.04081/) (2024)
- [Bi-Fact: A Bidirectional Factorization-based Evaluation of Intent Extraction from UI Trajectories](https://mlsanity.com/paper/2502.13149/) (2025)
- [ScreenAudit: Detecting Screen Reader Accessibility Errors in Mobile Apps Using Large Language Models](https://mlsanity.com/paper/2504.02110/) (2025)
- [IntentCUA: Learning Intent-level Representations for Skill Abstraction and Multi-Agent Planning in Computer-Use Agents](https://mlsanity.com/paper/2602.17049/) (2026)
- [Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach](https://mlsanity.com/paper/2509.22137/) (2025)
- [MacroBench: A Novel Testbed for Web Automation Scripts via Large Language Models](https://mlsanity.com/paper/2510.04363/) (2025)
- [Motif: Discovering and Automating Personal Web Workflows](https://mlsanity.com/paper/2607.10531/) (2026)

## Abstract (arXiv)

Macros are building block tasks of our everyday smartphone activity (e.g., "login", or "booking a flight"). Effectively extracting macros is important for understanding mobile interaction and enabling task automation. These macros are however difficult to extract at scale as they can be comprised of multiple steps yet hidden within programmatic components of mobile apps. In this paper, we introduce a novel approach based on Large Language Models (LLMs) to automatically extract semantically meaningful macros from both random and user-curated mobile interaction traces. The macros produced by our approach are automatically tagged with natural language descriptions and are fully executable. We conduct multiple studies to validate the quality of extracted macros, including user evaluation, comparative analysis against human-curated tasks, and automatic execution of these macros. These experiments and analyses show the effectiveness of our approach and the usefulness of extracted macros in various downstream applications.