mlsanityData to 7 Oct 2026

Language Control Diffusion: Efficiently Scaling through Space, Time, and Tasks

arXiv 2210.15629 ↗First posted 27 Oct 2022cs.LG cs.AI cs.CLcited by 20

On the map: Diffusion models → Discrete and text diffusion

Abstract (arXiv)

Training generalist agents is difficult across several axes, requiring us to deal with high-dimensional inputs (space), long horizons (time), and generalization to novel tasks. Recent advances with architectures have allowed for improved scaling along one or two of these axes, but are still computationally prohibitive to use. In this paper, we propose to address all three axes by leveraging Language to Control Diffusion models as a hierarchical planner conditioned on language (LCD). We effectively and efficiently scale diffusion models for planning in extended temporal, state, and task dimensions to tackle long horizon control problems conditioned on natural language instructions, as a step towards generalist agents. Comparing LCD with other state-of-the-art models on the CALVIN language robotics benchmark finds that LCD outperforms other SOTA methods in multi-task success rates, whilst improving inference speed over other comparable diffusion models by 3.3x~15x. We show that LCD can successfully leverage the unique strength of diffusion models to produce coherent long range plans while addressing their weakness in generating low-level details and control.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in Discrete and text diffusion, one of 14 directions in Diffusion models. Builds on 61 of its 85 references that are in the map. First posted on arXiv 27 Oct 2022.

Builds on

61 of 85 references.

References · 61

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  10. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  11. 2022Video Diffusion Modelscited by 2,280
  12. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
38 more
  1. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  4. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  5. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2022Inner Monologue: Embodied Reasoning through Planning with Language Modelscited by 1,057
  8. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  9. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  10. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  11. 2022A Generalist Agentcited by 802
  12. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  13. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  14. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  15. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  16. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  17. 2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
  18. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
  19. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  20. 2018A Survey on Deep Transfer Learningcited by 417
  21. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  22. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  23. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  24. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progresscited by 294
  25. 2019Causal Confusion in Imitation Learningcited by 286
  26. 2021A Survey of Embodied AI: From Simulators to Research Taskscited by 256
  27. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  28. 2015Contextual Markov Decision Processescited by 213
  29. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  30. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  31. 2019A Survey of Reinforcement Learning Informed by Natural Languagecited by 154
  32. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  33. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  34. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  35. 2017Grounded Language Learning in a Simulated 3D Worldcited by 100
  36. 2018Lipschitz Continuity in Model-based Reinforcement Learningcited by 90
  37. 2022Latent Plans for Task-Agnostic Offline Reinforcement Learningcited by 81
  38. 2018Learning to Understand Goal Specifications by Modelling Rewardcited by 80

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 8; 2025: 7; 2026: 4.

20 papers.

Sort:

By area:

  1. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  2. 2024DeeR-VLA: Dynamic Inference of Multimodal Large Language Models for Efficient Robot Executioncited by 116
  3. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  4. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  5. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 51
  6. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 42
  7. 2024Diffusion Transformer Policycited by 41
  8. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  9. 2024CleanDiffuser: An Easy-to-use Modularized Library for Diffusion Models in Decision Makingcited by 38
  10. 2024RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulationcited by 20
  11. 2025LUMOS: Language-Conditioned Imitation Learning with World Modelscited by 13
  12. 2026Language-Conditioned World Modeling for Visual Navigationcited by 9
8 more
  1. 2024DISCO: Language-Guided Manipulation with Diffusion Policies and Constrained Inpaintingcited by 8
  2. 2025Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policycited by 8
  3. 2026VPWEM: Non-Markovian Visuomotor Policy with Working and Episodic Memorycited by 6
  4. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 5
  5. 2026CF-VLA: Efficient Coarse-to-Fine Action Generation for Vision-Language-Action Policiescited by 4
  6. 2025PANDORA: Diffusion Policy Learning for Dexterous Robotic Piano Playingcited by 2
  7. 2026Online Self-Training for Co-Adaptation in Hierarchical Diffusion Policiescited by 1
  8. 2025SeFA-Policy: Fast and Accurate Visuomotor Policy Learning with Selective Flow Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Plan, Eliminate, and Track -- Language Models are Good Teachers for Embodied Agents
  2. 2024MALMM: Multi-Agent Large Language Models for Zero-Shot Robotics Manipulation
  3. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  4. 2023Conditionally Combining Robot Skills using Large Language Models
  5. 2023Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
  6. 2025PilotRL: Training Language Model Agents via Global Planning-Guided Progressive Reinforcement Learning
  7. 2025ReAcTree: Hierarchical LLM Agent Trees with Control Flow for Long-Horizon Task Planning
  8. 2026Embodied Task Planning via Graph-Informed Action Generation with Large Language Models