mlsanityData to 7 Oct 2026

Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition

arXiv 2307.14535 ↗First posted 26 Jul 2023cs.ROcited by 156

Not on the map. Why?

Cited by 31 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present a framework for robot skill acquisition, which 1) efficiently scale up data generation of language-labelled robot data and 2) effectively distills this data down into a robust multi-task language-conditioned visuo-motor policy. For (1), we use a large language model (LLM) to guide high-level planning, and sampling-based robot planners (e.g. motion or grasp samplers) for generating diverse and rich manipulation trajectories. To robustify this data-collection process, the LLM also infers a code-snippet for the success condition of each task, simultaneously enabling the data-collection process to detect failure and retry as well as the automatic labeling of trajectories with success/failure. For (2), we extend the diffusion policy single-task behavior-cloning approach to multi-task settings with language conditioning. Finally, we propose a new multi-task benchmark with 18 tasks across five domains to test long-horizon behavior, common-sense reasoning, tool-use, and intuitive physics. We find that our distilled policy successfully learned the robust retrying behavior in its data collection procedure, while improving absolute success rates by 33.2% on average across five domains. Code, data, and additional qualitative results are available on https://www.cs.columbia.edu/~huy/scalingup/.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). First posted on arXiv 26 Jul 2023.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 68; 2025: 50; 2026: 26.

156 papers.

Sort:

By area:

  1. 2024Octo: An Open-Source Generalist Robot Policycited by 1,427
  2. 2025GR00T N1: An Open Foundation Model for Generalist Humanoid Robotscited by 1,167
  3. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 814
  4. 2025CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Modelscited by 440
  5. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  6. 2023Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulationcited by 353
  7. 2023MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrationscited by 308
  8. 2024GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulationcited by 290
  9. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  10. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  11. 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 206
  12. 2025DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Controlcited by 191
38 more
  1. 2025DreamGen: Unlocking Generalization in Robot Learning through Video World Modelscited by 168
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  3. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  4. 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 120
  5. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  6. 2024DexMimicGen: Automated Data Generation for Bimanual Dexterous Manipulation via Imitation Learningcited by 104
  7. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 102
  8. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  9. 2024Large Language Models for Robotics: Opportunities, Challenges, and Perspectivescited by 88
  10. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  11. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  12. 2024Manipulate-Anything: Automating Real-World Robots using Vision-Language Modelscited by 82
  13. 2024General Flow as Foundation Affordance for Scalable Robot Learningcited by 80
  14. 2024DrEureka: Language Model Guided Sim-To-Real Transfercited by 78
  15. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  16. 2025Large VLM-based Vision-Language-Action Models for Robotic Manipulation: A Surveycited by 66
  17. 2024In-Context Imitation Learning via Next-Token Predictioncited by 65
  18. 2023Gen2Sim: Scaling up Robot Learning in Simulation with Generative Modelscited by 62
  19. 2025SpatialCoT: Advancing Spatial Reasoning through Coordinate Alignment and Chain-of-Thought for Embodied Task Planningcited by 62
  20. 2024Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Taskscited by 51
  21. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 51
  22. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  23. 2024GenSim2: Scaling Robot Data Generation with Multi-modal and Reasoning LLMscited by 48
  24. 2024SkillMimicGen: Automated Demonstration Generation for Efficient Skill Learning and Deploymentcited by 48
  25. 2024RACER: Rich Language-Guided Failure Recovery Policies for Imitation Learningcited by 46
  26. 2024Re-Mix: Optimizing Data Mixtures for Large Scale Imitation Learningcited by 44
  27. 2024Diffusion Transformer Policycited by 41
  28. 2025Articulate AnyMesh: Open-Vocabulary 3D Articulated Objects Modelingcited by 41
  29. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  30. 2024Potential Based Diffusion Motion Planningcited by 40
  31. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  32. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  33. 2024Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpaintingcited by 37
  34. 2023Interactive Task Planning with Language Modelscited by 36
  35. 2024Deep Generative Models in Robotics: A Survey on Learning from Multimodal Demonstrationscited by 35
  36. 2024AffordDP: Generalizable Diffusion Policy with Transferable Affordancecited by 35
  37. 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
  38. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32

50 most cited of 156 All 156 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Agentic Skill Discovery
  2. 2024Rethinking Mutual Information for Language Conditioned Skill Discovery on Imitation Learning
  3. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks
  4. 2023Bridging Language and Action: A Survey of Language-Conditioned Robot Manipulation
  5. 2023Conditionally Combining Robot Skills using Large Language Models
  6. 2024VLABench: A Large-Scale Benchmark for Language-Conditioned Robotics Manipulation with Long-Horizon Reasoning Tasks
  7. 2024Yell At Your Robot: Improving On-the-Fly from Language Corrections
  8. 2022Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models