# Interactive Program Synthesis for Modeling Collaborative Physical Activities from Narrated Demonstrations

arXiv: https://arxiv.org/abs/2509.24250 · HTML: https://mlsanity.com/paper/2509.24250/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 76 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Machine Theory of Mind](https://mlsanity.com/paper/1802.07740/) (2018)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [Agentic Large Language Models, a survey](https://mlsanity.com/paper/2503.23037/) (2025)
- [Scenic: A Language for Scenario Specification and Scene Generation](https://mlsanity.com/paper/1809.09310/) (2018)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [Scenic: A Language for Scenario Specification and Data Generation](https://mlsanity.com/paper/2010.06580/) (2020)
- [WaitGPT: Monitoring and Steering Conversational LLM Agent in Data Analysis with On-the-Fly Code Visualization](https://mlsanity.com/paper/2408.01703/) (2024)
- [Spellburst: A Node-based Interface for Exploratory Creative Coding with Natural Language Prompts](https://mlsanity.com/paper/2308.03921/) (2023)
- [Improving Steering and Verification in AI-Assisted Data Analysis with Interactive Task Decomposition](https://mlsanity.com/paper/2407.02651/) (2024)
- [Interactive Task and Concept Learning from Natural Language Instructions and GUI Demonstrations](https://mlsanity.com/paper/1909.00031/) (2019)
- [Reinforcement learning with human advice: a survey](https://mlsanity.com/paper/2005.11016/) (2020)
- [Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Tasks](https://mlsanity.com/paper/2210.04476/) (2022)
- [A Narration-based Reward Shaping Approach using Grounded Natural Language Commands](https://mlsanity.com/paper/1911.00497/) (2019)
- [VAL: Interactive Task Learning with GPT Dialog Parsing](https://mlsanity.com/paper/2310.01627/) (2023)
- [Reward Learning from Narrated Demonstrations](https://mlsanity.com/paper/1804.10692/) (2018)
- [SQLucid: Grounding Natural Language Database Queries with Interactive Explanations](https://mlsanity.com/paper/2409.06178/) (2024)
- [Falx: Synthesis-Powered Visualization Authoring](https://mlsanity.com/paper/2102.01024/) (2021)
- [Guided Reality: Generating Visually-Enriched AR Task Guidance with LLMs and Vision Models](https://mlsanity.com/paper/2508.03547/) (2025)
- [TutoAI: A Cross-domain Framework for AI-assisted Mixed-media Tutorial Creation on Physical Tasks](https://mlsanity.com/paper/2403.08049/) (2024)
- [DiLogics: Creating Web Automation Programs With Diverse Logics](https://mlsanity.com/paper/2308.05828/) (2023)

## Built on by (most cited first, 2 of 2)

- [InterPReT: Interactive Policy Restructuring and Training Enable Effective Imitation Learning from Laypersons](https://mlsanity.com/paper/2602.04213/) (2026)
- [Iterative Policy Refinement through Semantic Rollout Analysis](https://mlsanity.com/paper/2610.01652/) (2026)

## Abstract (arXiv)

Teaching systems physical tasks is a long standing goal in HCI, yet most prior work has focused on non collaborative physical activities. Collaborative tasks introduce added complexity, requiring systems to infer users assumptions about their teammates intent, which is an inherently ambiguous and dynamic process. This necessitates representations that are interpretable and correctable, enabling users to inspect and refine system behavior. We address this challenge by framing collaborative task learning as a program synthesis problem. Our system represents behavior as editable programs and uses narrated demonstrations, i.e. paired physical actions and natural language, as a unified modality for teaching, inspecting, and correcting system logic without requiring users to see or write code. The same modality is used for the system to communicate its learning to users. In a within subjects study, 20 users taught multiplayer soccer tactics to our system. 70 percent (14/20) of participants successfully refined learned programs to match their intent and 90 percent (18/20) found it easy to correct the programs. The study surfaced unique challenges in representing learning as programs and in enabling users to teach collaborative physical activities. We discuss these issues and outline mitigation strategies.