mlsanityData to 7 Oct 2026

CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character control

arXiv 2410.03441 ↗First posted 4 Oct 2024cs.CVcited by 103

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Motion diffusion models and Reinforcement Learning (RL) based control for physics-based simulations have complementary strengths for human motion generation. The former is capable of generating a wide variety of motions, adhering to intuitive control such as text, while the latter offers physically plausible motion and direct interaction with the environment. In this work, we present a method that combines their respective strengths. CLoSD is a text-driven RL physics-based controller, guided by diffusion generation for various tasks. Our key insight is that motion diffusion can serve as an on-the-fly universal planner for a robust RL controller. To this end, CLoSD maintains a closed-loop interaction between two modules -- a Diffusion Planner (DiP), and a tracking controller. DiP is a fast-responding autoregressive diffusion model, controlled by textual prompts and target locations, and the controller is a simple and robust motion imitator that continuously receives motion plans from DiP and provides feedback from the environment. CLoSD is capable of seamlessly performing a sequence of different tasks, including navigation to a goal location, striking an object with a hand or foot as specified in a text prompt, sitting down, and getting up. https://guytevet.github.io/CLoSD-page/

Key facts. Cited by 103 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 53 of its 61 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

53 of 61 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024PDP: Physics-Based Character Animation via Diffusion Policy3 Jun
  2. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion4 Apr
  3. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  4. 2023Diffusion-based Generation, Optimization, and Planning in 3D Scenes15 Jan

Other references · 49

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2022Human Motion Diffusion Modelcited by 1,062
  10. 2020Improved Techniques for Training Score-Based Generative Modelscited by 1,036
  11. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  12. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
37 more
  1. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  2. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  3. 2023Human Motion Diffusion as a Generative Priorcited by 354
  4. 2022EDGE: Editable Dance Generation From Musiccited by 284
  5. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  6. 2023Perpetual Humanoid Control for Real-time Simulated Avatarscited by 198
  7. 2022MotionBERT: A Unified Perspective on Learning Human Motion Representationscited by 188
  8. 2022TEACH: Temporal Action Composition for 3D Humanscited by 173
  9. 2023Universal Humanoid Motion Representations for Physics-Based Controlcited by 109
  10. 2019MCP: Learning Composable Hierarchical Control with Multiplicative Compositional Policiescited by 105
  11. 2023Synthesizing Physical Character-Scene Interactionscited by 102
  12. 2021Dynamics-Regulated Kinematic Policy for Egocentric Pose Estimationcited by 101
  13. 2024MaskedMimic: Unified Physics-Based Character Control Through Masked Motion Inpaintingcited by 101
  14. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  15. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  16. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  17. 2024Generating Human Interaction Motions in Scenes with Text Controlcited by 80
  18. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  19. 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
  20. 2024Flexible Motion In-betweening with Diffusion Modelscited by 69
  21. 2023Single Motion Diffusioncited by 65
  22. 2020Residual Force Control for Agile Human Behavior Imitation and Extended Motion Synthesiscited by 63
  23. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  24. 2024Taming Diffusion Probabilistic Models for Character Controlcited by 56
  25. 2023Interactive Character Control with Auto-Regressive Motion Diffusion Modelscited by 52
  26. 2024Generating Human Motion in 3D Scenes from Text Descriptionscited by 49
  27. 2023PADL: Language-Directed Physics-Based Character Controlcited by 47
  28. 2024THOR: Text to Human-Object Interaction Diffusion via Relation Interventioncited by 46
  29. 2023Hierarchical Planning and Control for Box Loco-Manipulationcited by 42
  30. 2023GenZI: Zero-Shot 3D Human-Scene Interaction Generationcited by 40
  31. 2020UniCon: Universal Neural Controller For Physics-based Character Motioncited by 39
  32. 2023Iterative Motion Editing with Natural Languagecited by 39
  33. 2023MAS: Multi-view Ancestral Sampling for 3D motion generation using 2D diffusioncited by 37
  34. 2024Monkey See, Monkey Do: Harnessing Self-attention in Motion Diffusion for Zero-shot Motion Transfercited by 30
  35. 2022Embodied Scene-aware Human Pose Estimationcited by 28
  36. 2023InsActor: Instruction-driven Physics-based Characterscited by 23
  37. 2024SuperPADL: Scaling Language-Directed Physics-Based Control with Progressive Supervised Distillationcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 43; 2026: 56.

103 papers.

Sort:

By area:

  1. 2025BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusioncited by 128
  2. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  3. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  4. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  5. 2025InterMimic: Towards Universal Whole-Body Control for Physics-Based Human-Object Interactionscited by 67
  6. 2025AnyTop: Character Animation Diffusion with Any Topologycited by 37
  7. 2025InterAct: Advancing Large-Scale Versatile 3D Human-Object Interaction Generationcited by 31
  8. 2025Absolute Coordinates Make Motion Generation Easycited by 27
  9. 2025A Survey on Human Interaction Motion Generationcited by 25
  10. 2025UniPhys: Unified Planner and Controller with Diffusion for Flexible Physics-Based Character Controlcited by 24
  11. 2025Towards Immersive Human-X Interaction: A Real-Time Framework for Physically Plausible Motion Synthesiscited by 20
  12. 2025Dance Like a Chicken: Low-Rank Stylization for Human Motion Diffusioncited by 16
38 more
  1. 2025HOIDiNi: Human-Object Interaction through Diffusion Noise Optimizationcited by 16
  2. 2024Mimicking-Bench: A Benchmark for Generalizable Humanoid-Scene Interaction Learning via Human Mimickingcited by 14
  3. 2025SMP: Reusable Score-Matching Motion Priors for Physics-Based Character Controlcited by 14
  4. 2025PRIMAL: Physically Reactive and Interactive Motor Model for Avatar Learningcited by 13
  5. 2025Reconstructing 4D Spatial Intelligence: A Surveycited by 13
  6. 2025DreamControl: Human-Inspired Whole-Body Humanoid Control for Scene Interaction via Guided Diffusioncited by 13
  7. 2025Human-Object Interaction via Automatically Designed VLM-Guided Motion Policycited by 10
  8. 2026InterPrior: Scaling Generative Control for Physics-Based Human-Object Interactionscited by 10
  9. 2025Motion Generation: A Survey of Generative Approaches and Benchmarkscited by 8
  10. 2025ModSkill: Physical Character Skill Modularizationcited by 7
  11. 2025ViBES: A Conversational Agent with Behaviorally-Intelligent 3D Virtual Bodycited by 6
  12. 2025Interact2Ar: Full-Body Human-Human Interaction Generation via Autoregressive Diffusion Modelscited by 6
  13. 2026SafeMo: Linguistically Grounded Unlearning for Trustworthy Text-to-Motion Generationcited by 6
  14. 2025DreamPolicy: A Unified World-model Policy for Scalable Humanoid Locomotioncited by 5
  15. 2025UniAct: Unified Motion Generation and Action Streaming for Humanoid Robotscited by 5
  16. 2026UMO: Unified In-Context Learning Unlocks Motion Foundation Model Priorscited by 5
  17. 2026ARDY: Autoregressive Diffusion with Hybrid Representation for Interactive Human Motion Generationcited by 5
  18. 2025Half-Physics: Enabling Kinematic 3D Human Model with Physical Interactionscited by 4
  19. 2025Moving by Looking: Towards Vision-Driven Avatar Motion Generationcited by 4
  20. 2025InterAgent: Physics-based Multi-agent Command Execution via Diffusion on Interaction Graphscited by 4
  21. 2026ZEST: Zero-shot Embodied Skill Transfer for Athletic Robot Controlcited by 4
  22. 2026TeamHOI: Learning a Unified Policy for Cooperative Human-Object Interactions with Any Team Sizecited by 4
  23. 2026ActionPlan: Future-Aware Streaming Motion Synthesis via Frame-Level Action Planningcited by 4
  24. 2026SCRIPT: Scalable Diffusion Policy with Multi-stage Training for Language-driven Physics-Based Humanoid Controlcited by 4
  25. 2025SyncSDE: A Probabilistic Framework for Diffusion Synchronizationcited by 3
  26. 2025Autonomous Human-Robot Interaction via Operator Imitationcited by 3
  27. 2025SimGenHOI: Physically Realistic Whole-Body Humanoid-Object Interaction via Generative Modeling and Reinforcement Learningcited by 3
  28. 2024SMGDiff: Soccer Motion Generation using diffusion probabilistic modelscited by 2
  29. 2025Automatic Generation of Aerobatic Flight in Complex Environments via Diffusion Modelscited by 2
  30. 2025Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motionscited by 2
  31. 2025MoLingo: Motion-Language Alignment for Text-to-Human Motion Generationcited by 2
  32. 2026HSImul3R: Physics-in-the-Loop Reconstruction of Simulation-Ready Human-Scene Interactionscited by 2
  33. 2026PhyGile: Physics-Prefix Guided Motion Generation for Agile General Humanoid Motion Trackingcited by 2
  34. 2026RoMo: A Large-Scale, Richly Organized Dataset and Semantic Taxonomy for Human Motion Generationcited by 2
  35. 2025SimDiff: Simulator-constrained Diffusion Model for Physically Plausible Motion Generationcited by 1
  36. 2025Controllable Long-term Motion Generation with Extended Joint Targetscited by 1
  37. 2025Prime and Reach: Synthesising Body Motion for Gaze-Primed Object Reachcited by 1
  38. 2025Solver-Independent Automated Problem Formulation via LLMs for High-Cost Simulation-Driven Designcited by 1

50 most cited of 103 All 103 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Diffuse-CLoC: Guided Diffusion for Physics-based Character Look-ahead Control
  2. 2025Scalable Motion In-betweening via Diffusion and Physics-Based Character Adaptation
  3. 2024ReinDiffuse: Crafting Physically Plausible Motions with Reinforced Diffusion Model
  4. 2025Pixel Motion Diffusion is What We Need for Robot Control
  5. 2025Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulation
  6. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model
  7. 2026A Single Diffusion-Policy Controller for Multi-Task Block Pushing with Zero-Shot Sim-to-Real Transfer
  8. 2025Beyond Human Demonstrations: Diffusion-Based Reinforcement Learning to Generate Data for VLA Training