mlsanityData to 7 Oct 2026

Executing your Commands via Motion Diffusion in Latent Space

arXiv 2212.04048 ↗First posted 8 Dec 2022cs.CV cs.GRcited by 567

On the map: Diffusion models → Video generation

Abstract (arXiv)

We study a challenging task, conditional human motion generation, which produces plausible human motion sequences according to various conditional inputs, such as action classes or textual descriptors. Since human motions are highly diverse and have a property of quite different distribution from conditional modalities, such as textual descriptors in natural languages, it is hard to learn a probabilistic mapping from the desired conditional modality to the human motion sequences. Besides, the raw motion data from the motion capture system might be redundant in sequences and contain noises; directly modeling the joint distribution over the raw motion sequences and conditional modalities would need a heavy computational overhead and might result in artifacts introduced by the captured noises. To learn a better representation of the various human motion sequences, we first design a powerful Variational AutoEncoder (VAE) and arrive at a representative and low-dimensional latent code for a human motion sequence. Then, instead of using a diffusion model to establish the connections between the raw motion sequences and the conditional inputs, we perform a diffusion process on the motion latent space. Our proposed Motion Latent-based Diffusion model (MLD) could produce vivid motion sequences conforming to the given conditional inputs and substantially reduce the computational overhead in both the training and inference stages. Extensive experiments on various human motion generation tasks demonstrate that our MLD achieves significant improvements over the state-of-the-art methods among extensive human motion generation tasks, with two orders of magnitude faster than previous diffusion models on raw motion sequences.

Key facts. Cited by 567 papers in the map (data through 7 Oct 2026). Placed in Video generation, one of 14 directions in Diffusion models. Builds on 57 of its 82 references that are in the map. First posted on arXiv 8 Dec 2022.

Builds on

57 of 82 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Imagen Video: High Definition Video Generation with Diffusion Models5 Oct
  2. 2022Human Motion Diffusion Model29 Sep
  3. 2022FLAME: Free-form Language-based Motion Synthesis & Editing1 Sep
  4. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model31 Aug

Other references · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  5. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  6. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  7. 2020Denoising Diffusion Implicit Modelscited by 9,655
  8. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  9. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  10. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  11. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  12. 2022Classifier-Free Diffusion Guidancecited by 5,717
38 more
  1. 2017Improved Training of Wasserstein GANscited by 2,223
  2. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  3. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2021Variational Diffusion Modelscited by 1,145
  6. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  7. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  8. 20213D Shape Generation and Completion through Point-Voxel Diffusioncited by 516
  9. 2022GeoDiff: a Geometric Diffusion Model for Molecular Conformation Generationcited by 511
  10. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  11. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  12. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  13. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  14. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  15. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  16. 2016The KIT Motion-Language Datasetcited by 334
  17. 2021Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecastingcited by 332
  18. 2021HuMoR: 3D Human Motion Model for Robust Pose Estimationcited by 319
  19. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  20. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  21. 2021Robust Motion In-betweeningcited by 223
  22. 2021Character Controllers Using Motion VAEscited by 182
  23. 2022Embodied Hands: Modeling and Capturing Hands and Bodies Togethercited by 179
  24. 2021Synthesis of Compositional Animations from Textual Descriptionscited by 177
  25. 2020We are More than Our Joints: Predicting how 3D Bodies Movecited by 115
  26. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  27. 2017Text2Action: Generative Adversarial Synthesis from Language to Actioncited by 106
  28. 2021Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agentscited by 106
  29. 2021Encoder-decoder with Multi-level Attention for 3D Human Shape and Pose Estimationcited by 62
  30. 2017Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networkscited by 61
  31. 2022Implicit Neural Representations for Variable Length Human Motion Generationcited by 61
  32. 2021Single-Shot Motion Completion with Transformercited by 59
  33. 2022MoDi: Unconditional Motion Synthesis from Diverse Datacited by 59
  34. 2022BARC: Learning to Regress 3D Dog Shape from Images by Exploiting Breed Informationcited by 51
  35. 2018Human Motion Modeling using DVGANscited by 45
  36. 2021SportsCap: Monocular 3D Human Motion Capture and Fine-grained Understanding in Challenging Sports Videoscited by 40
  37. 2020Perpetual Motion: Generating Unbounded Human Motioncited by 36
  38. 2021ChallenCap: Monocular 3D Capture of Challenging Human Performances using Multi-Modal Referencescited by 26

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 78; 2024: 150; 2025: 184; 2026: 155.

567 papers.

Sort:

By area:

  1. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  2. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  3. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  4. 2023Human Motion Diffusion as a Generative Priorcited by 354
  5. 2023GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Modelscited by 255
  6. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  7. 2023Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representationcited by 225
  8. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  9. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  10. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  11. 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 194
  12. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
38 more
  1. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  2. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  3. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  4. 2023MMM: Generative Masked Motion Modelcited by 129
  5. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  6. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 116
  7. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  8. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  9. 2023Controllable Human-Object Interaction Synthesiscited by 107
  10. 2023FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editingcited by 105
  11. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  12. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  13. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  14. 2023Human Motion Generation: A Surveycited by 95
  15. 2025Being-H0: Vision-Language-Action Pretraining from Large-Scale Human Videoscited by 94
  16. 2023Generative Image Dynamicscited by 93
  17. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  18. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  19. 2023HOI-Diff: Text-Driven Synthesis of 3D Human-Object Interactions using Diffusion Modelscited by 89
  20. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  21. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  22. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  23. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  24. 2024Text2HOI: Text-guided 3D Motion Generation for Hand-Object Interactioncited by 84
  25. 2025MotionStreamer: Streaming Motion Generation via Diffusion-based Autoregressive Model in Causal Latent Spacecited by 82
  26. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  27. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  28. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  29. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  30. 2024D4M: Dataset Distillation via Disentangled Diffusion Modelcited by 76
  31. 2023PhysHOI: Physics-Based Imitation of Dynamic Human-Object Interactioncited by 72
  32. 2024Flexible Motion In-betweening with Diffusion Modelscited by 69
  33. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 67
  34. 2023Priority-Centric Human Motion Generation in Discrete Latent Spacecited by 66
  35. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  36. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  37. 2024Analysis of Classifier-Free Guidance Weight Schedulerscited by 64
  38. 2023AvatarGPT: All-in-One Framework for Motion Understanding, Planning, Generation and Beyondcited by 63

50 most cited of 567 All 567 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023AMD: Autoregressive Motion Diffusion
  2. 2025Back to Basics: Motion Representation Matters for Human Motion Generation Using Diffusion Model
  3. 2022Diffusion Motion: Generate Text-Guided 3D Human Motion by Diffusion Model
  4. 2022Unifying Human Motion Synthesis and Style Transfer with Denoising Diffusion Probabilistic Models
  5. 2024Multi-Condition Latent Diffusion Network for Scene-Aware Neural Human Motion Prediction
  6. 2022BeLFusion: Latent Diffusion for Behavior-Driven Human Motion Prediction
  7. 2023Synthesizing Long-Term Human Motions with Diffusion Models via Coherent Sampling
  8. 2023LaMD: Latent Motion Diffusion for Image-Conditional Video Generation