mlsanityData to 14 Oct 2026

UINav: A Practical Approach to Train On-Device Automation Agents

arXiv 2312.10170 ↗First posted 15 Dec 2023cs.HC cs.AIcited by 5

Not on the map. Why?

Abstract (arXiv)

Automation systems that can autonomously drive application user interfaces to complete user tasks are of great benefit, especially when users are situationally or permanently impaired. Prior automation systems do not produce generalizable models while AI-based automation agents work reliably only in simple, hand-crafted applications or incur high computation costs. We propose UINav, a demonstration-based approach to train automation agents that fit mobile devices, yet achieving high success rates with modest numbers of demonstrations. To reduce the demonstration overhead, UINav uses a referee model that provides users with immediate feedback on tasks where the agent fails, and automatically augments human demonstrations to increase diversity in training data. Our evaluation shows that with only 10 demonstrations UINav can achieve 70% accuracy, and that with enough demonstrations it can surpass 90% accuracy.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 22 of its 46 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

22 of 46 references.

References · 22

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  3. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  4. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 319
  5. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  6. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 181
  7. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  8. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 95
  9. 2021AndroidEnv: A Reinforcement Learning Platform for Androidcited by 90
  10. 2022A data-driven approach for learning to control computerscited by 88
  11. 2018A Survey on Semantic Parsingcited by 62
  12. 2022Spotlight: Mobile UI Understanding using Vision-Language Models with a Focuscited by 62
10 more
  1. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  2. 2018Learning to Navigate the Webcited by 60
  3. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 58
  4. 2021Few-Shot Semantic Parsing with Language Models Trained On Codecited by 50
  5. 2020ActionBert: Leveraging User Actions for Semantic Understanding of User Interfacescited by 43
  6. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  7. 2020Object Detection for Graphical User Interface: Old Fashioned or Deep Learning or a Combination?cited by 29
  8. 2023Lexi: Self-Supervised Learning of the UI Languagecited by 10
  9. 2020Translating Video Recordings of Mobile App Usages into Replayable Scenarioscited by 9
  10. 2021Learning UI Navigation through Demonstrations composed of Macro Actionscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 2.

5 papers.

  1. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 260
  2. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  3. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  4. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  5. 2024Parameter Efficient Reinforcement Learning from Human Feedbackcited by 7