# Cocobo: Exploring Large Language Models as the Engine for End-User Robot Programming

arXiv: https://arxiv.org/abs/2407.20712 · HTML: https://mlsanity.com/paper/2407.20712/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 50 references that are in the map.
- First posted on arXiv 30 Jul 2024.

## Builds on (most cited first, 15 of 15)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [ChatGPT for Robotics: Design Principles and Model Abilities](https://mlsanity.com/paper/2306.17582/) (2023)
- [AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Prompts](https://mlsanity.com/paper/2110.01691/) (2021)
- [The Programmer's Assistant: Conversational Interaction with a Large Language Model for Software Development](https://mlsanity.com/paper/2302.07080/) (2023)
- [Graphologue: Exploring Large Language Model Responses with Interactive Diagrams](https://mlsanity.com/paper/2305.11473/) (2023)
- ["What It Wants Me To Say": Bridging the Abstraction Gap Between End-User Programmers and Code-Generating Large Language Models](https://mlsanity.com/paper/2304.06597/) (2023)
- [In-IDE Code Generation from Natural Language: Promise and Challenges](https://mlsanity.com/paper/2101.11149/) (2021)
- [What is it like to program with artificial intelligence?](https://mlsanity.com/paper/2208.06213/) (2022)
- [A Survey on End-User Robot Programming](https://mlsanity.com/paper/2105.01757/) (2021)
- [Evaluating Methods for End-User Creation of Robot Task Plans](https://mlsanity.com/paper/1811.02690/) (2018)
- [Sketching Robot Programs On the Fly](https://mlsanity.com/paper/2302.03088/) (2023)

## Built on by (most cited first, 6 of 6)

- [AutoMisty: A Multi-Agent LLM Framework for Automated Code Generation in the Misty Social Robot](https://mlsanity.com/paper/2503.06791/) (2025)
- [GenComUI: Exploring Generative Visual Aids as Medium to Support Task-Oriented Human-Robot Communication](https://mlsanity.com/paper/2502.10678/) (2025)
- [SkillComposer: Learning Reusable Skills for Natural-Language Robot Programming](https://mlsanity.com/paper/2608.14944/) (2026)
- [Bootstrapping Human-Like Planning via LLMs](https://mlsanity.com/paper/2506.22604/) (2025)
- [Feasibility of AI-Assisted Programming for End-User Development](https://mlsanity.com/paper/2512.05666/) (2025)
- [VoCa: Designing Speech-Canvas Interaction for Voice-Based Conversational Agents](https://mlsanity.com/paper/2610.04706/) (2026)

## Abstract (arXiv)

End-user development allows everyday users to tailor service robots or applications to their needs. One user-friendly approach is natural language programming. However, it encounters challenges such as an expansive user expression space and limited support for debugging and editing, which restrict its application in end-user programming. The emergence of large language models (LLMs) offers promising avenues for the translation and interpretation between human language instructions and the code executed by robots, but their application in end-user programming systems requires further study. We introduce Cocobo, a natural language programming system with interactive diagrams powered by LLMs. Cocobo employs LLMs to understand users' authoring intentions, generate and explain robot programs, and facilitate the conversion between executable code and flowchart representations. Our user study shows that Cocobo has a low learning curve, enabling even users with zero coding experience to customize robot programs successfully.