Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation
Not on the map. Why?
Abstract (arXiv)
Choreographic motion generation poses unique challenges for AI, demanding precise semantic control over complex, temporally structured, and expressive full-body dynamics. While existing models can synthesize motion from music, they remain largely black boxes. Conversely, attempting to condition generation on both text and music frequently leads to modality collapse, where dense acoustic rhythms overwhelm sparse semantic text prompts, destroying user controllability. To resolve this spatial-temporal conflict, we propose STREAM (Structural-Temporal Rhythmic Energy-based Attention for Motion), a modality-decoupled diffusion transformer. STREAM strictly separates conditioning pathways: global text semantics dictate the kinematic structure via Adaptive Layer Normalization (AdaLN), while a novel Bimodal Energy-Based Attention Module (BEAM) routes these features to the musical beat without overwriting the semantics. We further introduce Motorica++, a newly curated dataset enriched with domain-specific dance vocabulary and frame-level semantic annotations from existing Motorica dataset. Additionally, to rigorously quantify zero-shot editability, we propose the Exchange Evaluation Protocol and Editable Dance Score (EDS). Through extensive experiments, STREAM achieves state-of-the-art alignment between motion and music while perfectly preserving choreographic semantics, positioning AI not merely as a reactive synthesizer, but as a controllable, collaborative partner for artistic direction. The source code and datasets are available at https://github.com/SeongJong-Yoo/STREAM.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 73 references that are in the map. First posted on arXiv 22 Jun 2026.
Builds on
47 of 73 references.
References · 47
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020Denoising Diffusion Probabilistic Modelscited by 21,076
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
- 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
- 2022Scalable Diffusion Models with Transformerscited by 6,054
- 2022Classifier-Free Diffusion Guidancecited by 5,847
- 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,214
- 2022Human Motion Diffusion Modelcited by 1,080
- 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 575
- 2020Jukebox: A Generative Model for Musiccited by 542
- 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 485
35 more
- 2020Hopfield Networks is All You Needcited by 475
- 2016The KIT Motion-Language Datasetcited by 338
- 2022EDGE: Editable Dance Generation From Musiccited by 287
- 2023ReMoDiffuse: Retrieval-Augmented Motion Diffusion Modelcited by 287
- 2021How to Train Your Energy-Based Modelscited by 274
- 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 227
- 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 220
- 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 212
- 2023TMR: Text-to-Motion Retrieval Using Contrastive 3D Human Motion Synthesiscited by 198
- 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 190
- 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 88
- 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
- 2020Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learningcited by 81
- 2023NIFTY: Neural Object Interaction Fields for Guided Human Motion Synthesiscited by 78
- 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
- 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 67
- 2024MotionFix: Text-Driven 3D Human Motion Editingcited by 64
- 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 61
- 2025GENMO: A GENeralist Model for Human MOtioncited by 49
- 2022MultiAct: Long-Term 3D Human Motion Generation from Multiple Action Labelscited by 45
- 2023Iterative Motion Editing with Natural Languagecited by 40
- 2024Learning Iterative Reasoning through Energy Diffusioncited by 36
- 2025Energy-Based Transformers are Scalable Learners and Thinkerscited by 31
- 2023Energy-Based Cross Attention for Bayesian Context Update in Text-to-Image Diffusion Modelscited by 30
- 2024EnergyMoGen: Compositional Human Motion Generation with Energy-Based Diffusion Model in Latent Spacecited by 30
- 2025Absolute Coordinates Make Motion Generation Easycited by 29
- 2024POPDG: Popular 3D Dance Generation with PopDanceSetcited by 24
- 2024Beat-It: Beat-Synchronized Multi-Condition 3D Dance Generationcited by 24
- 2024HUMOS: Human Motion Model Conditioned on Body Shapecited by 20
- 2024UniMuMo: Unified Text, Music and Motion Generationcited by 19
- 2025Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representationcited by 8
- 2023EnchantDance: Unveiling the Potential of Music-Driven Dance Movementcited by 7
- 2025DanceEditor: Towards Iterative Editable Music-driven Dance Generation with Open-Vocabulary Descriptionscited by 6
- 2025MDD: A Dataset for Text-and-Music Conditioned Duet Dance Generationcited by 4
- 2024Choreographing the Digital Canvas: A Machine Learning Approach to Artistic Performancecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026Listen to Rhythm, Choose Movements: Autoregressive Multimodal Dance Generation via Diffusion and Mamba with Decoupled Dance Dataset
- 2025MotionRAG-Diff: A Retrieval-Augmented Diffusion Framework for Long-Term Music-to-Dance Generation
- 2025MEGADance: Mixture-of-Experts Architecture for Genre-Aware 3D Dance Generation
- 2024LM2D: Lyrics- and Music-Driven Dance Synthesis
- 2025Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation
- 2023LongDanceDiff: Long-term Dance Generation with Conditional Diffusion Model
- 2026TokenDance: Token-to-Token Music-to-Dance Generation with Bidirectional Mamba
- 2025GACA-DiT: Diffusion-based Dance-to-Music Generation with Genre-Adaptive Rhythm and Context-Aware Alignment