# Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations

arXiv: https://arxiv.org/abs/1909.00031 · HTML: https://mlsanity.com/paper/1909.00031/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Aug 2019.

## Built on by (most cited first, 26 of 26)

- [Screen2Words: Automatic Mobile UI Summarization with Multimodal Learning](https://mlsanity.com/paper/2108.03353/) (2021)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [Enabling Conversational Interaction with Mobile UI using Large Language Models](https://mlsanity.com/paper/2209.08655/) (2022)
- [VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping](https://mlsanity.com/paper/2304.07810/) (2023)
- [StoryBuddy: A Human-AI Collaborative Chatbot for Parent-Child Interactive Storytelling with Flexible Parental Involvement](https://mlsanity.com/paper/2202.06205/) (2022)
- [Data Formulator: AI-powered Concept-driven Visualization Authoring](https://mlsanity.com/paper/2309.10094/) (2023)
- [Multimodal Conversational AI: A Survey of Datasets and Approaches](https://mlsanity.com/paper/2205.06907/) (2022)
- [VAL: Interactive Task Learning with GPT Dialog Parsing](https://mlsanity.com/paper/2310.01627/) (2023)
- [MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation](https://mlsanity.com/paper/2410.13757/) (2024)
- [Automatic Macro Mining from Interaction Traces at Scale](https://mlsanity.com/paper/2310.07023/) (2023)
- [SQLucid: Grounding Natural Language Database Queries with Interactive Explanations](https://mlsanity.com/paper/2409.06178/) (2024)
- [CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning](https://mlsanity.com/paper/2503.03743/) (2025)
- [DiLogics: Creating Web Automation Programs With Diverse Logics](https://mlsanity.com/paper/2308.05828/) (2023)
- [DrawTalking: Building Interactive Worlds by Sketching and Speaking](https://mlsanity.com/paper/2401.05631/) (2024)
- [Modeling Distinct Human Interaction in Web Agents](https://mlsanity.com/paper/2602.17588/) (2026)
- [TagRuler: Interactive Tool for Span-Level Data Programming by Demonstration](https://mlsanity.com/paper/2106.12767/) (2021)
- [AI for Low-Code for AI](https://mlsanity.com/paper/2305.20015/) (2023)
- [Tutorly: Turning Programming Videos Into Apprenticeship Learning Environments with LLMs](https://mlsanity.com/paper/2405.12946/) (2024)
- [Explorer: Robust Collection of Interactable GUI Elements](https://mlsanity.com/paper/2504.09352/) (2025)
- [Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations](https://mlsanity.com/paper/2509.24250/) (2025)
- [Understanding User Experiences of Computer Use Agents: Design Space and Opportunities for Building Agent UX Prototypes](https://mlsanity.com/paper/2510.04452/) (2025)
- [A Bottom-Up End-User Intelligent Assistant Approach to Empower Gig Workers against AI Inequality](https://mlsanity.com/paper/2204.13842/) (2022)
- [Interactively Diagnosing Errors in a Semantic Parser](https://mlsanity.com/paper/2407.06400/) (2024)
- [Log2Plan: An Adaptive GUI Automation Framework Integrated with Task Mining Approach](https://mlsanity.com/paper/2509.22137/) (2025)
- [Motif: Discovering and Automating Personal Web Workflows](https://mlsanity.com/paper/2607.10531/) (2026)
- [Richard: Voice-First Mobile Interaction for Persistent Tasks](https://mlsanity.com/paper/2609.39976/) (2026)

## Abstract (arXiv)

Natural language programming is a promising approach to enable end users to instruct new tasks for intelligent agents. However, our formative study found that end users would often use unclear, ambiguous or vague concepts when naturally instructing tasks in natural language, especially when specifying conditionals. Existing systems have limited support for letting the user teach agents new concepts or explaining unclear concepts. In this paper, we describe a new multi-modal domain-independent approach that combines natural language programming and programming-by-demonstration to allow users to first naturally describe tasks and associated conditions at a high level, and then collaborate with the agent to recursively resolve any ambiguities or vagueness through conversations and demonstrations. Users can also define new procedures and concepts by demonstrating and referring to contents within GUIs of existing mobile apps. We demonstrate this approach in PUMICE, an end-user programmable agent that implements this approach. A lab study with 10 users showed its usability.