mlsanityData to 7 Oct 2026

PlayFusion: Skill Acquisition via Diffusion from Language-Annotated Play

arXiv 2312.04549 ↗First posted 7 Dec 2023cs.RO cs.AI cs.CVcited by 48

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Learning from unstructured and uncurated data has become the dominant paradigm for generative approaches in language and vision. Such unstructured and unguided behavior data, commonly known as play, is also easier to collect in robotics but much more difficult to learn from due to its inherently multimodal, noisy, and suboptimal nature. In this paper, we study this problem of learning goal-directed skill policies from unstructured play data which is labeled with language in hindsight. Specifically, we leverage advances in diffusion models to learn a multi-task diffusion model to extract robotic skills from play data. Using a conditional denoising diffusion process in the space of states and actions, we can gracefully handle the complexity and multimodality of play data and generate diverse and interesting robot behaviors. To make diffusion models more useful for skill learning, we encourage robotic agents to acquire a vocabulary of skills by introducing discrete bottlenecks into the conditional behavior generation process. In our experiments, we demonstrate the effectiveness of our approach across a wide variety of environments in both simulation and the real world. Results visualizations and videos at https://play-fusion.github.io

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 74 references that are in the map. First posted on arXiv 7 Dec 2023.

Builds on

53 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  2. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  4. 2022StructDiffusion: Language-Guided Creation of Physically-Valid Structures using Unseen Objects8 Nov
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 48

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
36 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  3. 2023Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardwarecited by 1,490
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2017Hindsight Experience Replaycited by 1,096
  6. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  7. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  9. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  10. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  12. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  13. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  14. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  15. 2023Learning Universal Policies via Text-Guided Video Generationcited by 506
  16. 2016The Option-Critic Architecturecited by 433
  17. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  18. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  19. 2019Learning Latent Plans from Playcited by 310
  20. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  21. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  22. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 220
  23. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  24. 2019Goal-conditioned Imitation Learningcited by 159
  25. 2022What Matters in Language Conditioned Robotic Imitation Learning over Unstructured Datacited by 137
  26. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  27. 2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
  28. 2021Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Taskscited by 75
  29. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  30. 2020Learning Robot Skills with Temporal Variational Inferencecited by 54
  31. 2022Retrieval-Augmented Reinforcement Learningcited by 53
  32. 2021Vector Quantized Models for Planningcited by 47
  33. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  34. 2019Dynamics-aware Embeddingscited by 34
  35. 2021Hierarchical Neural Dynamic Policiescited by 17
  36. 2022Cross-Domain Transfer via Semantic Skill Imitationcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 25; 2025: 16; 2026: 5.

48 papers.

Sort:

By area:

  1. 2024RDT-1B: a Diffusion Foundation Model for Bimanual Manipulationcited by 670
  2. 20243D-VLA: A 3D Vision-Language-Action Generative World Modelcited by 320
  3. 2024Behavior Generation with Latent Actionscited by 159
  4. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  5. 2024Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goalscited by 131
  6. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 128
  7. 2025Unified Vision-Language-Action Modelcited by 128
  8. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  9. 2025Dita: Scaling Diffusion Transformer for Generalist Vision-Language-Action Policycited by 73
  10. 2024Streaming Diffusion Policy: Fast Policy Synthesis with Variable Noise Diffusion Modelscited by 51
  11. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 49
  12. 2025Can We Detect Failures Without Failure Data? Uncertainty-Aware Runtime Failure Detection for Imitation Learning Policiescited by 49
36 more
  1. 2024DexDiffuser: Generating Dexterous Grasps with Diffusion Modelscited by 46
  2. 2024Diffusion Transformer Policycited by 41
  3. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 39
  4. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 34
  5. 2024Diffusion Predictive Control with Constraintscited by 32
  6. 2024Don't Start from Scratch: Behavioral Refinement via Interpolant-based Policy Diffusioncited by 21
  7. 2024Render and Diffuse: Aligning Image and Action Spaces for Diffusion-based Behaviour Cloningcited by 19
  8. 2024Generative Image as Action Modelscited by 18
  9. 2025AMPLIFY: Actionless Motion Priors for Robot Learning from Videoscited by 18
  10. 2024Diffusion Policy Attacker: Crafting Adversarial Attacks for Diffusion-based Policiescited by 17
  11. 2025XR-1: Towards Versatile Vision-Language-Action Models via Learning Unified Vision-Motion Representationscited by 15
  12. 2025Demystifying Diffusion Policies: Action Memorization and Simple Lookup Table Alternativescited by 9
  13. 2024A Robotic Skill Learning System Built Upon Diffusion Policies and Foundation Modelscited by 8
  14. 2024SPIRE: Synergistic Planning, Imitation, and Reinforcement Learning for Long-Horizon Manipulationcited by 8
  15. 2025Grounding Actions in Camera Space: Observation-Centric Vision-Language-Action Policycited by 8
  16. 2026Chain of World: World Model Thinking in Latent Motioncited by 8
  17. 2023LHManip: A Dataset for Long-Horizon Language-Grounded Manipulation Tasks in Cluttered Tabletop Environmentscited by 7
  18. 2024Generative AI Agents in Autonomous Machines: A Safety Perspectivecited by 7
  19. 2024Creative Problem Solving in Large Language and Vision Models -- What Would it Take?cited by 6
  20. 2025Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLAcited by 4
  21. 2024D-Cubed: Latent Diffusion Trajectory Optimisation for Dexterous Deformable Manipulationcited by 3
  22. 2025Multimodal Diffusion Forcing for Forceful Manipulationcited by 3
  23. 2024Meta-Control: Automatic Model-based Control Synthesis for Heterogeneous Robot Skillscited by 2
  24. 2025Enhancing Diffusion Policy with Classifier-Free Guidance for Temporal Robotic Taskscited by 2
  25. 2024Precise Pick-and-Place using Score-Based Diffusion Networkscited by 1
  26. 2024Implicit Contact Diffuser: Sequential Contact Reasoning with Latent Point Cloud Diffusioncited by 1
  27. 2025Controllable Latent Diffusion for Traffic Simulationcited by 1
  28. 2026WestWorld: A Knowledge-Encoded Scalable Trajectory World Model for Diverse Robotic Systemscited by 1
  29. 2024Learning Generalizable 3D Manipulation With 10 Demonstrationscited by 0
  30. 2025Spooky Action at a Distance: Normalization Layers Enable Side-Channel Spatial Communicationcited by 0
  31. 2025UniPrototype: Humn-Robot Skill Learning with Uniform Prototypescited by 0
  32. 2025EVLP:Learning Unified Embodied Vision-Language Planner with Reinforced Supervised Fine-Tuningcited by 0
  33. 2025SeFA-Policy: Fast and Accurate Visuomotor Policy Learning with Selective Flow Alignmentcited by 0
  34. 2026Primary-Fine Decoupling for Action Generation in Robotic Imitationcited by 0
  35. 2026From Noise to Control: Parameterized Diffusion Policiescited by 0
  36. 2026Multi-Granularity Language-Guided Imitation Learning via Instruction Decompositioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026DSD: Learning Diverse and Reusable Motor Skills via Diffusion Skill Discovery
  2. 2025X-Diffusion: Training Diffusion Policies on Cross-Embodiment Human Demonstrations
  3. 2023Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
  4. 2024Composing Diffusion Policies for Few-shot Learning of Movement Trajectories
  5. 2026Learning Diffusion Policy from Primitive Skills for Robot Manipulation
  6. 2024EXTRACT: Efficient Policy Learning by Extracting Transferable Robot Skills from Offline Data
  7. 2025S2-Diffusion: Generalizing from Instance-level to Category-level Skills in Robot Manipulation
  8. 2023MimicPlay: Long-Horizon Imitation Learning by Watching Human Play