# Demo2Code: From Summarizing Demonstrations to Synthesizing Code via Extended Chain-of-Thought

arXiv: https://arxiv.org/abs/2305.16744 · HTML: https://mlsanity.com/paper/2305.16744/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 7 of its 69 references that are in the map.
- First posted on arXiv 26 May 2023.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 7 of 7)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Least-to-Most Prompting Enables Complex Reasoning in Large Language Models](https://mlsanity.com/paper/2205.10625/) (2022)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language](https://mlsanity.com/paper/2204.00598/) (2022)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)

## Built on by (most cited first, 30 of 39)

- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [RoCo: Dialectic Multi-Robot Collaboration with Large Language Models](https://mlsanity.com/paper/2307.04738/) (2023)
- [WorldCoder, a Model-Based LLM Agent: Building World Models by Writing Code and Interacting with the Environment](https://mlsanity.com/paper/2402.12275/) (2024)
- [Vision Language Models are In-Context Value Learners](https://mlsanity.com/paper/2411.04549/) (2024)
- [DrEureka: Language Model Guided Sim-To-Real Transfer](https://mlsanity.com/paper/2406.01967/) (2024)
- [If LLM Is the Wizard, Then Code Is the Wand: A Survey on How Code Empowers Large Language Models to Serve as Intelligent Agents](https://mlsanity.com/paper/2401.00812/) (2024)
- [SteP: Stacked LLM Policies for Web Actions](https://mlsanity.com/paper/2310.03720/) (2023)
- [A Large Language Model-Driven Reward Design Framework via Dynamic Feedback for Reinforcement Learning](https://mlsanity.com/paper/2410.14660/) (2024)
- [Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions](https://mlsanity.com/paper/2410.12773/) (2024)
- [Empowering Large Language Model Agents through Action Learning](https://mlsanity.com/paper/2402.15809/) (2024)
- [SwarmBrain: Embodied agent for real-time strategy game StarCraft II via large language models](https://mlsanity.com/paper/2401.17749/) (2024)
- [Eurekaverse: Environment Curriculum Generation via Large Language Models](https://mlsanity.com/paper/2411.01775/) (2024)
- [ReCode: Unify Plan and Action for Universal Granularity Control](https://mlsanity.com/paper/2510.23564/) (2025)
- [From Pixels to Predicates: Learning Symbolic World Models via Pretrained Vision-Language Models](https://mlsanity.com/paper/2501.00296/) (2024)
- [VLMimic: Vision Language Models are Visual Imitation Learner for Fine-grained Actions](https://mlsanity.com/paper/2410.20927/) (2024)
- [Modeling Others' Minds as Code](https://mlsanity.com/paper/2510.01272/) (2025)
- [BYOC: Personalized Few-Shot Classification with Co-Authored Class Descriptions](https://mlsanity.com/paper/2310.06111/) (2023)
- [O3D: Offline Data-driven Discovery and Distillation for Sequential Decision-Making with Large Language Models](https://mlsanity.com/paper/2310.14403/) (2023)
- [Natural Language as Policies: Reasoning for Coordinate-Level Embodied Control with LLMs](https://mlsanity.com/paper/2403.13801/) (2024)
- [Reconstruct, Practice, Go Real: Guided Self-Improvement for Embodied Agents](https://mlsanity.com/paper/2610.02204/) (2026)
- [Knowing What Not to Do: Leverage Language Model Insights for Action Space Pruning in Multi-agent Reinforcement Learning](https://mlsanity.com/paper/2405.16854/) (2024)
- [Skill-Aligned Fairness in Multi-Agent Learning for Collaboration in Healthcare](https://mlsanity.com/paper/2508.18708/) (2025)
- [A Few Words Go a Long Way: Language Guided Robot Policy Synthesis](https://mlsanity.com/paper/2607.23784/) (2026)
- [DocuMint: Docstring Generation for Python using Small Language Models](https://mlsanity.com/paper/2405.10243/) (2024)
- [RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Models](https://mlsanity.com/paper/2406.03757/) (2024)
- [Learning and Transferring Closed-Loop Robot Software](https://mlsanity.com/paper/2609.19906/) (2026)
- [Program Machine Policy: Addressing Long-Horizon Tasks by Integrating Program Synthesis and State Machines](https://mlsanity.com/paper/2311.15960/) (2023)
- [Creating and Repairing Robot Programs in Open-World Domains](https://mlsanity.com/paper/2410.18893/) (2024)
- [Long-Horizon Visual Imitation Learning via Plan and Code Reflection](https://mlsanity.com/paper/2509.05368/) (2025)
- [Intuitive Programming, Adaptive Task Planning, and Dynamic Role Allocation in Human-Robot Collaboration](https://mlsanity.com/paper/2511.08732/) (2025)

## Abstract (arXiv)

Language instructions and demonstrations are two natural ways for users to teach robots personalized tasks. Recent progress in Large Language Models (LLMs) has shown impressive performance in translating language instructions into code for robotic tasks. However, translating demonstrations into task code continues to be a challenge due to the length and complexity of both demonstrations and code, making learning a direct mapping intractable. This paper presents Demo2Code, a novel framework that generates robot task code from demonstrations via an extended chain-of-thought and defines a common latent specification to connect the two. Our framework employs a robust two-stage process: (1) a recursive summarization technique that condenses demonstrations into concise specifications, and (2) a code synthesis approach that expands each function recursively from the generated specifications. We conduct extensive evaluation on various robot task benchmarks, including a novel game benchmark Robotouille, designed to simulate diverse cooking tasks in a kitchen environment. The project's website is available at https://portal-cornell.github.io/demo2code/