# Pretrained Diffusion Models for Unified Human Motion Synthesis

arXiv: https://arxiv.org/abs/2212.02837 · HTML: https://mlsanity.com/paper/2212.02837/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 6 Dec 2022.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 30 of 40)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Deep Unsupervised Learning using Nonequilibrium Thermodynamics](https://mlsanity.com/paper/1503.03585/) (2015)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models](https://mlsanity.com/paper/2112.10741/) (2021)
- [Decoupled Weight Decay Regularization](https://mlsanity.com/paper/1711.05101/) (2017)
- [RePaint: Inpainting using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2201.09865/) (2022)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [On the Continuity of Rotation Representations in Neural Networks](https://mlsanity.com/paper/1812.07035/) (2018)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [Improved Techniques for Training Score-Based Generative Models](https://mlsanity.com/paper/2006.09011/) (2020)
- [Diffusion-LM Improves Controllable Text Generation](https://mlsanity.com/paper/2205.14217/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Learning 3D Human Dynamics from Video](https://mlsanity.com/paper/1812.01601/) (2018)
- [HybrIK: A Hybrid Analytical-Neural Inverse Kinematics Solution for 3D Human Pose and Shape Estimation](https://mlsanity.com/paper/2011.14672/) (2020)
- [BABEL: Bodies, Action and Behavior with English Labels](https://mlsanity.com/paper/2106.09696/) (2021)
- [Language2Pose: Natural Language Grounded Pose Forecasting](https://mlsanity.com/paper/1907.01108/) (2019)
- [Robust Motion In-betweening](https://mlsanity.com/paper/2102.04942/) (2021)
- [FLAME: Free-form Language-based Motion Synthesis & Editing](https://mlsanity.com/paper/2209.00349/) (2022)

## Built on by (most cited first, 26 of 26)

- [Synthetic Data from Diffusion Models Improves ImageNet Classification](https://mlsanity.com/paper/2304.08466/) (2023)
- [Guided Motion Diffusion for Controllable Human Motion Synthesis](https://mlsanity.com/paper/2305.12577/) (2023)
- [Optimizing Diffusion Noise Can Serve As Universal Motion Priors](https://mlsanity.com/paper/2312.11994/) (2023)
- [FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation](https://mlsanity.com/paper/2212.03741/) (2022)
- [Interactive Character Control with Auto-Regressive Motion Diffusion Models](https://mlsanity.com/paper/2306.00416/) (2023)
- [Generative Proxemics: A Prior for 3D Social Interaction from Images](https://mlsanity.com/paper/2306.09337/) (2023)
- [Hierarchical Generation of Human-Object Interactions with Diffusion Probabilistic Models](https://mlsanity.com/paper/2310.02242/) (2023)
- [Probabilistic Human Mesh Recovery in 3D Scenes from Egocentric Views](https://mlsanity.com/paper/2304.06024/) (2023)
- [Towards Open Domain Text-Driven Synthesis of Multi-Person Motions](https://mlsanity.com/paper/2405.18483/) (2024)
- [MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controls](https://mlsanity.com/paper/2407.21136/) (2024)
- [Do Generated Data Always Help Contrastive Learning?](https://mlsanity.com/paper/2403.12448/) (2024)
- [Freetalker: Controllable Speech and Text-Driven Gesture Generation Based on Diffusion Models for Enhanced Speaker Naturalness](https://mlsanity.com/paper/2401.03476/) (2024)
- [Synth2: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings](https://mlsanity.com/paper/2403.07750/) (2024)
- [Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motion](https://mlsanity.com/paper/2404.11375/) (2024)
- [UnifiedGesture: A Unified Gesture Synthesis Model for Multiple Skeletons](https://mlsanity.com/paper/2309.07051/) (2023)
- [3DiFACE: Diffusion-based Speech-driven 3D Facial Animation and Editing](https://mlsanity.com/paper/2312.00870/) (2023)
- [SocialGen: Modeling Multi-Human Social Interaction with Language Models](https://mlsanity.com/paper/2503.22906/) (2025)
- [Towards Detailed Text-to-Motion Synthesis via Basic-to-Advanced Hierarchical Diffusion Model](https://mlsanity.com/paper/2312.10960/) (2023)
- [Bridging the Gap: Enhancing the Utility of Synthetic Data via Post-Processing Techniques](https://mlsanity.com/paper/2305.10118/) (2023)
- [AAMDM: Accelerated Auto-regressive Motion Diffusion Model](https://mlsanity.com/paper/2401.06146/) (2023)
- [Synthetic to Authentic: Transferring Realism to 3D Face Renderings for Boosting Face Recognition](https://mlsanity.com/paper/2407.07627/) (2024)
- [3DiFACE: Synthesizing and Editing Holistic 3D Facial Animation](https://mlsanity.com/paper/2509.26233/) (2025)
- [DF-3DFace: One-to-Many Speech Synchronized 3D Face Animation with Diffusion](https://mlsanity.com/paper/2310.05934/) (2023)
- [MixDiff: Mixing Natural and Synthetic Images for Robust Self-Supervised Representations](https://mlsanity.com/paper/2406.12368/) (2024)
- [Biomechanics-Guided Residual Approach to Generalizable Human Motion Generation and Estimation](https://mlsanity.com/paper/2503.06151/) (2025)
- [Multimodal Generative AI with Autoregressive LLMs for Human Motion Understanding and Generation: A Way Forward](https://mlsanity.com/paper/2506.03191/) (2025)

## Abstract (arXiv)

Generative modeling of human motion has broad applications in computer animation, virtual reality, and robotics. Conventional approaches develop separate models for different motion synthesis tasks, and typically use a model of a small size to avoid overfitting the scarce data available in each setting. It remains an open question whether developing a single unified model is feasible, which may 1) benefit the acquirement of novel skills by combining skills learned from multiple tasks, and 2) help in increasing the model capacity without overfitting by combining multiple data sources. Unification is challenging because 1) it involves diverse control signals as well as targets of varying granularity, and 2) motion datasets may use different skeletons and default poses. In this paper, we present MoFusion, a framework for unified motion synthesis. MoFusion employs a Transformer backbone to ease the inclusion of diverse control signals via cross attention, and pretrains the backbone as a diffusion model to support multi-granularity synthesis ranging from motion completion of a body part to whole-body motion generation. It uses a learnable adapter to accommodate the differences between the default skeletons used by the pretraining and the fine-tuning data. Empirical results show that pretraining is vital for scaling the model size without overfitting, and demonstrate MoFusion's potential in various tasks, e.g., text-to-motion, motion completion, and zero-shot mixing of multiple control signals. Project page: urlhttps://ofa-sys.github.io/MoFusion/.