mlsanityData to 7 Oct 2026

MotionCLIP: Exposing Human Motion Generation to CLIP Space

arXiv 2203.08063 ↗First posted 15 Mar 2022cs.CV cs.GRcited by 406

Not on the map. Why?

Cited by 69 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce MotionCLIP, a 3D human motion auto-encoder featuring a latent embedding that is disentangled, well behaved, and supports highly semantic textual descriptions. MotionCLIP gains its unique power by aligning its latent space with that of the Contrastive Language-Image Pre-training (CLIP) model. Aligning the human motion manifold to CLIP space implicitly infuses the extremely rich semantic knowledge of CLIP into the manifold. In particular, it helps continuity by placing semantically similar motions close to one another, and disentanglement, which is inherited from the CLIP-space structure. MotionCLIP comprises a transformer-based motion auto-encoder, trained to reconstruct motion while being aligned to its text label's position in CLIP-space. We further leverage CLIP's unique visual understanding and inject an even stronger signal through aligning motion to rendered frames in a self-supervised manner. We show that although CLIP has never seen the motion domain, MotionCLIP offers unprecedented text-to-motion abilities, allowing out-of-domain actions, disentangled editing, and abstract language specification. For example, the text prompt "couch" is decoded into a sitting down motion, due to lingual similarity, and the prompt "Spiderman" results in a web-swinging-like solution that is far from seen during training. In addition, we show how the introduced latent space can be leveraged for motion interpolation, editing and recognition.

Key facts. Cited by 406 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 15 Mar 2022.

Builds on

32 of 50 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  6. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  7. 2019Expressive Body Capture: 3D Hands, Face, and Body from a Single Imagecited by 1,600
  8. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  9. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  10. 2014NICE: Non-linear Independent Components Estimationcited by 1,052
  11. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  12. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
20 more
  1. 2019NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understandingcited by 530
  2. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  3. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  4. 2020Action2Motion: Conditioned Generation of 3D Human Motionscited by 438
  5. 2021CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fieldscited by 369
  6. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  7. 2016The KIT Motion-Language Datasetcited by 334
  8. 2021Text2Mesh: Text-Driven Neural Stylization for Meshescited by 332
  9. 2021BABEL: Bodies, Action and Behavior with English Labelscited by 261
  10. 2019Language2Pose: Natural Language Grounded Pose Forecastingcited by 257
  11. 2021CLIP-Forge: Towards Zero-Shot Text-to-Shape Generationcited by 254
  12. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 223
  13. 2016Deep Unsupervised Clustering with Gaussian Mixture Variational Autoencoderscited by 219
  14. 2021CLIPDraw: Exploring Text-to-Drawing Synthesis through Language-Image Encoderscited by 186
  15. 2020Unpaired Motion Style Transfer from Video to Animationcited by 94
  16. 2021Multi-Person 3D Motion Prediction with Multi-Range Transformerscited by 63
  17. 2017Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networkscited by 61
  18. 2021Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structurecited by 53
  19. 2017Fine-graind Image Classification via Combining Vision and Languagecited by 51
  20. 2021MUGL: Large Scale Multi Person Conditional Action Generation with Locomotioncited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 27; 2023: 85; 2024: 114; 2025: 110; 2026: 70.

406 papers.

Sort:

By area:

  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2022Human Motion Diffusion Modelcited by 1,062
  3. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  4. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  5. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  6. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  7. 2022Latent-NeRF for Shape-Guided Generation of 3D Shapes and Texturescited by 506
  8. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  9. 2023Human Motion Diffusion as a Generative Priorcited by 354
  10. 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 282
  11. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  12. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
38 more
  1. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  2. 2023Teaching CLIP to Count to Tencited by 194
  3. 2023InterDiff: Generating 3D Human-Object Interactions with Physics-Informed Diffusioncited by 181
  4. 2022TEACH: Temporal Action Composition for 3D Humanscited by 173
  5. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  6. 2023GestureDiffuCLIP: Gesture Diffusion Model with CLIP Latentscited by 156
  7. 2022SlotFormer: Unsupervised Visual Dynamics Simulation with Object-Centric Modelscited by 146
  8. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  9. 2023CG-HOI: Contact-Guided 3D Human-Object Interaction Generationcited by 137
  10. 2023MMM: Generative Masked Motion Modelcited by 129
  11. 2022IMos: Intent-Driven Full-Body Motion Synthesis for Human-Object Interactionscited by 121
  12. 2023Delta Denoising Scorecited by 120
  13. 2023AttT2M: Text-Driven Human Motion Generation with Multi-Perspective Attention Mechanismcited by 116
  14. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  15. 2023Synthesizing Diverse Human Motions in 3D Indoor Scenescited by 109
  16. 2024Nymeria: A Massive Collection of Multimodal Egocentric Daily Motion in the Wildcited by 109
  17. 2023Controllable Human-Object Interaction Synthesiscited by 107
  18. 2023FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editingcited by 105
  19. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
  20. 2024CLoSD: Closing the Loop between Simulation and Diffusion for multi-task character controlcited by 103
  21. 2023DreamHuman: Animatable 3D Avatars from Textcited by 102
  22. 2023Human Motion Generation: A Surveycited by 95
  23. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  24. 2023Unified Human-Scene Interaction via Prompted Chain-of-Contactscited by 91
  25. 2024MotionLLM: Understanding Human Behaviors from Human Motions and Videoscited by 89
  26. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  27. 2023Language Conditioned Traffic Generationcited by 86
  28. 2022ZeroEGGS: Zero-shot Example-based Gesture Generation from Speechcited by 83
  29. 2022Compositional Human-Scene Interaction Synthesis with Semantic Controlcited by 82
  30. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  31. 2024Move as You Say, Interact as You Can: Language-guided Human Motion Generation with Scene Affordancecited by 79
  32. 2024Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregressioncited by 78
  33. 2023CLIP goes 3D: Leveraging Prompt Tuning for Language Grounded 3D Recognitioncited by 77
  34. 2024Seamless Human Motion Composition with Blended Positional Encodingscited by 77
  35. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  36. 2022UDE: A Unified Driving Engine for Human Motion Generationcited by 67
  37. 2025Go to Zero: Towards Zero-shot Motion Generation with Million-scale Datacited by 67
  38. 2023Priority-Centric Human Motion Generation in Discrete Latent Spacecited by 66

50 most cited of 406 All 406 (JSON)