# EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Space

arXiv: https://arxiv.org/abs/2412.14706 · HTML: https://mlsanity.com/paper/2412.14706/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Video generation, one of 14 directions in Diffusion models.
- Builds on 14 of its 15 references that are in the map.
- First posted on arXiv 19 Dec 2024.

## Placed in

- Diffusion models / [Video generation](https://mlsanity.com/diffusion/video-generation/)

## Builds on (most cited first, 14 of 14)

- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [AMASS: Archive of Motion Capture as Surface Shapes](https://mlsanity.com/paper/1904.03278/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [Human Motion Diffusion as a Generative Prior](https://mlsanity.com/paper/2303.01418/) (2023)
- [The KIT Motion-Language Dataset](https://mlsanity.com/paper/1607.03827/) (2016)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [ReMoDiffuse: Retrieval-Augmented Motion Diffusion Model](https://mlsanity.com/paper/2304.01116/) (2023)
- [TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesis](https://mlsanity.com/paper/2305.00976/) (2023)
- [TEACH: Temporal Action Composition for 3D Humans](https://mlsanity.com/paper/2209.04066/) (2022)
- [Motion Mamba: Efficient and Long Sequence Motion Generation](https://mlsanity.com/paper/2403.07487/) (2024)
- [FineMoGen: Fine-Grained Spatio-Temporal Motion Generation and Editing](https://mlsanity.com/paper/2312.15004/) (2023)
- [Multi-Track Timeline Control for Text-Driven 3D Human Motion Generation](https://mlsanity.com/paper/2401.08559/) (2024)
- [Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2306.09869/) (2023)

## Built on by (most cited first, 30 of 30)

- [Generative Trajectory Stitching through Diffusion Composition](https://mlsanity.com/paper/2503.05153/) (2025)
- [Absolute Coordinates Make Motion Generation Easy](https://mlsanity.com/paper/2505.19377/) (2025)
- [ReAlign: Text-to-Motion Generation via Step-Aware Reward-Guided Alignment](https://mlsanity.com/paper/2511.19217/) (2025)
- [FrankenMotion: Part-level Human Motion Generation and Composition](https://mlsanity.com/paper/2601.10909/) (2026)
- [Causal Motion Diffusion Models for Autoregressive Motion Generation](https://mlsanity.com/paper/2602.22594/) (2026)
- [SentiAvatar: Towards Expressive and Interactive Digital Humans](https://mlsanity.com/paper/2604.02908/) (2026)
- [MotionHiFlow: Text-to-motion via hierarchical flow matching](https://mlsanity.com/paper/2604.23264/) (2026)
- [Decoupled Generative Modeling for Human-Object Interaction Synthesis](https://mlsanity.com/paper/2512.19049/) (2025)
- [Cross-Axis Feature Fusion with Joint-Wise Motion Difference Prediction for Text-Based 3D Human Motion Editing](https://mlsanity.com/paper/2606.01014/) (2026)
- [Topology-Agnostic Animal Motion Generation from Text Prompt](https://mlsanity.com/paper/2512.10352/) (2025)
- [Compositional Visual Planning via Inference-Time Diffusion Scaling](https://mlsanity.com/paper/2603.02646/) (2026)
- [Not All Frames Are Equal: Complexity-Aware Masked Motion Generation via Motion Spectral Descriptors](https://mlsanity.com/paper/2603.29655/) (2026)
- [ClusterStyle: Modeling Intra-Style Diversity with Prototypical Clustering for Stylized Motion Generation](https://mlsanity.com/paper/2512.02453/) (2025)
- [ReactMotion: Generating Reactive Listener Motions from Speaker Utterance](https://mlsanity.com/paper/2603.15083/) (2026)
- [UMo: Unified Sparse Motion Modeling for Real-Time Co-Speech Avatars](https://mlsanity.com/paper/2605.14731/) (2026)
- [DeMoGen: Towards Decompositional Human Motion Generation with Energy-Based Diffusion Models](https://mlsanity.com/paper/2512.22324/) (2025)
- [InterEdit: Navigating Text-Guided 3D Dyadic Human Motion Editing](https://mlsanity.com/paper/2603.13082/) (2026)
- [Controllable Image Generation with Composed Parallel Token Prediction](https://mlsanity.com/paper/2604.05730/) (2026)
- [Motion-Adapter: A Diffusion Model Adapter for Text-to-Motion Generation of Compound Actions](https://mlsanity.com/paper/2604.16135/) (2026)
- [Marrying Text-to-Motion Generation with Skeleton-Based Action Recognition](https://mlsanity.com/paper/2604.17090/) (2026)
- [Multi-scale Coarse-to-fine Modeling for Test-time Human Motion Control](https://mlsanity.com/paper/2605.14935/) (2026)
- [MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation](https://mlsanity.com/paper/2605.18956/) (2026)
- [Sparse Compositional Flow Matching by geometric assembly from motion primitives](https://mlsanity.com/paper/2605.23341/) (2026)
- [Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation](https://mlsanity.com/paper/2606.22726/) (2026)
- [InterCMDM: Block-Causal Diffusion for Autoregressive Human Interaction Generation](https://mlsanity.com/paper/2607.01743/) (2026)
- [ComplexMimic: Human-Scene Interaction Imitation in Complex 3D Environments](https://mlsanity.com/paper/2607.02034/) (2026)
- [GenPrior: Unleashing Text-to-Motion Generative Priors for Zero-Shot Skeleton-based Action Recognition](https://mlsanity.com/paper/2608.02236/) (2026)
- [MoVT: Video-Augmented Motion Tokenizer for Text-to-Motion Generation](https://mlsanity.com/paper/2609.14965/) (2026)
- [MotionSpaceFlow: Representation-Aware Flow Matching in Direct Motion Space](https://mlsanity.com/paper/2609.34190/) (2026)
- [BiMoGen: Bidirectional Motion-Text Generation via Unified Masked Discrete Diffusion](https://mlsanity.com/paper/2609.35407/) (2026)

## Abstract (arXiv)

Diffusion models, particularly latent diffusion models, have demonstrated remarkable success in text-driven human motion generation. However, it remains challenging for latent diffusion models to effectively compose multiple semantic concepts into a single, coherent motion sequence. To address this issue, we propose EnergyMoGen, which includes two spectrums of Energy-Based Models: (1) We interpret the diffusion model as a latent-aware energy-based model that generates motions by composing a set of diffusion models in latent space; (2) We introduce a semantic-aware energy model based on cross-attention, which enables semantic composition and adaptive gradient descent for text embeddings. To overcome the challenges of semantic inconsistency and motion distortion across these two spectrums, we introduce Synergistic Energy Fusion. This design allows the motion latent diffusion model to synthesize high-quality, complex motions by combining multiple energy terms corresponding to textual descriptions. Experiments show that our approach outperforms existing state-of-the-art models on various motion generation tasks, including text-to-motion generation, compositional motion generation, and multi-concept motion generation. Additionally, we demonstrate that our method can be used to extend motion datasets and improve the text-to-motion task.