mlsanityData to 7 Oct 2026

Enabling Conversational Interaction with Mobile UI using Large Language Models

arXiv 2209.08655 ↗First posted 18 Sep 2022cs.HC cs.AIcited by 66

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Conversational agents show the promise to allow users to interact with mobile devices using language. However, to perform diverse UI tasks with natural language, developers typically need to create separate datasets and models for each specific task, which is expensive and effort-consuming. Recently, pre-trained large language models (LLMs) have been shown capable of generalizing to various downstream tasks when prompted with a handful of examples from the target task. This paper investigates the feasibility of enabling versatile conversational interactions with mobile UIs using a single LLM. We designed prompting techniques to adapt an LLM to mobile UIs. We experimented with four important modeling tasks that address various scenarios in conversational interaction. Our method achieved competitive performance on these challenging tasks without requiring dedicated datasets and training, offering a lightweight and generalizable approach to enable language-based mobile interaction.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Builds on 34 of its 75 references that are in the map. First posted on arXiv 18 Sep 2022.

Builds on

34 of 75 references.

References · 34

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2022Emergent Abilities of Large Language Modelscited by 2,483
  11. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
22 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  3. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  4. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  5. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
  6. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 265
  7. 2022CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilitiescited by 245
  8. 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
  9. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 119
  10. 2022Rationale-Augmented Ensembles in Language Modelscited by 111
  11. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  12. 2022ScreenQA: Large-Scale Question-Answer Pairs over Mobile App Screenshotscited by 65
  13. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  14. 2021Screen Recognition: Creating Accessibility Metadata for Mobile Applications from Pixelscited by 57
  15. 2021Grounding 'Grounding' in NLPcited by 44
  16. 2022How to Prompt? Opportunities and Challenges of Zero- and Few-Shot Learning for Human-AI Interaction in Creative Applications of Generative Modelscited by 43
  17. 2021Screen2Vec: Semantic Embedding of GUI Screens and GUI Componentscited by 41
  18. 2021Screen Parsing: Towards Reverse Engineering of UI Models from Screenshotscited by 32
  19. 2022Learning to Denoise Raw Mobile UI Layouts for Improving Datasets at Scalecited by 30
  20. 2019Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrationscited by 26
  21. 2021VUT: Versatile UI Transformer for Multi-Modal Multi-Task User Interface Modelingcited by 26
  22. 2018Mapping Natural Language Commands to Web Elementscited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 25; 2024: 12; 2025: 18; 2026: 10.

66 papers.

Sort:

By area:

  1. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  2. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  3. 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
  4. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 180
  5. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  6. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  7. 2022Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environmentscited by 57
  8. 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
  9. 2023TPTU: Large Language Model-based AI Agents for Task Planning and Tool Usagecited by 54
  10. 2023Leveraging Large Language Models to Power Chatbots for Collecting User Self-Reported Datacited by 48
  11. 2024LAVE: LLM-Powered Agent Assistance and Language Augmentation for Video Editingcited by 47
  12. 2023TPTU-v2: Boosting Task Planning and Tool Usage of Large Language Model-based Agents in Real-world Systemscited by 39
38 more
  1. 2024LlamaTouch: A Faithful and Scalable Testbed for Mobile UI Task Automationcited by 36
  2. 2023Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interactioncited by 35
  3. 2024MELTing point: Mobile Evaluation of Language Transformerscited by 35
  4. 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
  5. 2023AXNav: Replaying Accessibility Tests from Natural Languagecited by 31
  6. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 31
  7. 2023A Zero-Shot Language Agent for Computer Control with Structured Reflectioncited by 28
  8. 2025A Survey of Personalization: From RAG to Agentcited by 26
  9. 2023ChoiceMates: Supporting Unfamiliar Online Decision-Making with Multi-Agent Conversational Interactionscited by 22
  10. 2025Morae: Proactively Pausing UI Agents for User Choicescited by 18
  11. 2023ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversationscited by 17
  12. 2024Large Language Models as Software Components: A Taxonomy for LLM-Integrated Applicationscited by 17
  13. 2025Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Reviewcited by 16
  14. 2025MobileSteward: Integrating Multiple App-Oriented Agents with Self-Evolution to Automate Cross-App Instructionscited by 16
  15. 2025Mobile-Bench-v2: A More Realistic and Comprehensive Benchmark for VLM-based Mobile Agentscited by 16
  16. 2023UI Layout Generation with LLMs Guided by UI Grammarcited by 15
  17. 2025Classroom Simulacra: Building Contextual Student Generative Agents in Online Education for Learning Behavioral Simulationcited by 15
  18. 2023Automatic Macro Mining from Interaction Traces at Scalecited by 12
  19. 2025Advancing Language Multi-Agent Learning with Credit Re-Assignment for Interactive Environment Generalizationcited by 11
  20. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Studycited by 11
  21. 2024From Interaction to Impact: Towards Safer AI Agents Through Understanding and Evaluating Mobile UI Operation Impactscited by 10
  22. 2025Evaluating the Sensitivity of LLMs to Prior Contextcited by 9
  23. 2025LLM-Explorer: Towards Efficient and Affordable LLM-based Exploration for Mobile Appscited by 8
  24. 2023"What's important here?": Opportunities and Challenges of Using LLMs in Retrieving Information from Web Interfacescited by 6
  25. 2023DiLogics: Creating Web Automation Programs With Diverse Logicscited by 5
  26. 2024A Human-Computer Collaborative Tool for Training a Single Large Language Model Agent into a Network through Few Examplescited by 5
  27. 2025Generative AI for Self-Adaptive Systems: State of the Art and Research Roadmapcited by 5
  28. 2023MenuCraft: Interactive Menu System Design with Large Language Modelscited by 4
  29. 2025Agent-Initiated Interaction in Phone UI Automationcited by 4
  30. 2025CORE: Reducing UI Exposure in Mobile Agents via Collaboration Between Cloud and Local LLMscited by 4
  31. 2026SecAgent: Efficient Mobile GUI Agent with Semantic Contextcited by 4
  32. 2026Executable Agentic Memory for GUI Agentcited by 4
  33. 2023Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniquescited by 3
  34. 2024CoLay: Controllable Layout Generation through Multi-conditional Latent Diffusioncited by 3
  35. 2023Referring to Screen Texts with Voice Assistantscited by 2
  36. 2023Large Language Models to the Rescue: Reducing the Complexity in Scientific Workflow Development Using ChatGPTcited by 2
  37. 2023InstructPipe: Generating Visual Blocks Pipelines with Human Instructions and LLMscited by 2
  38. 2023Talk2Care: Facilitating Asynchronous Patient-Provider Communication with Large-Language-Modelcited by 1

50 most cited of 66 All 66 (JSON)