mlsanityData to 7 Oct 2026

AI Choreographer: Music Conditioned 3D Dance Generation with AIST++

arXiv 2101.08779 ↗First posted 21 Jan 2021cs.CV cs.GR cs.MMcited by 479

Not on the map. Why?

Cited by 90 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present AIST++, a new multi-modal dataset of 3D dance motion and music, along with FACT, a Full-Attention Cross-modal Transformer network for generating 3D dance motion conditioned on music. The proposed AIST++ dataset contains 5.2 hours of 3D dance motion in 1408 sequences, covering 10 dance genres with multi-view videos with known camera poses -- the largest dataset of this kind to our knowledge. We show that naively applying sequence models such as transformers to this dataset for the task of music conditioned 3D motion generation does not produce satisfactory 3D motion that is well correlated with the input music. We overcome these shortcomings by introducing key changes in its architecture design and supervision: FACT model involves a deep cross-modal transformer block with full-attention that is trained to predict N future motions. We empirically show that these changes are key factors in generating long sequences of realistic dance motion that are well-attuned to the input music. We conduct extensive experiments on AIST++ with user studies, where our method outperforms recent state-of-the-art methods both qualitatively and quantitatively.

Key facts. Cited by 479 papers in the map (data through 7 Oct 2026). Builds on 46 of its 99 references that are in the map. First posted on arXiv 21 Jan 2021.

Builds on

46 of 99 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019AMASS: Archive of Motion Capture as Surface Shapescited by 1,208
  5. 2018OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,161
  6. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  7. 2017Dense-Captioning Events in Videoscited by 789
  8. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  9. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  10. 2019VIBE: Video Inference for Human Body Pose and Shape Estimationcited by 615
  11. 2015Structural-RNN: Deep Learning on Spatio-Temporal Graphscited by 234
  12. 2018End-to-End Dense Video Captioning with Masked Transformercited by 214
34 more
  1. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 210
  2. 2019Learning Individual Styles of Conversational Gesturecited by 190
  3. 2015Sequence to Sequence -- Video to Textcited by 157
  4. 2018Human Motion Prediction via Spatio-Temporal Inpaintingcited by 114
  5. 2020Multi-modal Dense Video Captioningcited by 113
  6. 2018LPCNet: Improving Neural Speech Synthesis Through Linear Predictioncited by 109
  7. 2018Neural Baby Talkcited by 107
  8. 2017Deep representation learning for human motion prediction and classificationcited by 97
  9. 2019Structured Prediction Helps 3D Human Motion Modellingcited by 96
  10. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  11. 2019Cross View Fusion for 3D Human Pose Estimationcited by 90
  12. 2015Describing Videos by Exploiting Temporal Structurecited by 88
  13. 2018Neural Kinematic Networks for Unsupervised Motion Retargettingcited by 85
  14. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
  15. 2020Foley Music: Learning to Generate Music from Videoscited by 82
  16. 2020Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learningcited by 81
  17. 2017Learning Human Motion Models for Long-term Predictionscited by 67
  18. 2018Weakly Supervised Dense Event Captioning in Videoscited by 58
  19. 2018Leveraging Weakly Supervised Data to Improve End-to-End Speech-to-Text Translationcited by 57
  20. 2018Action-Agnostic Human Pose Forecastingcited by 56
  21. 2015Video Paragraph Captioning Using Hierarchical Recurrent Neural Networkscited by 54
  22. 2017Benchmarking and Error Diagnosis in Multi-Instance Pose Estimationcited by 48
  23. 2017Audio to Body Dynamicscited by 48
  24. 2019Imitation Learning for Human Pose Predictioncited by 44
  25. 2020Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audiocited by 33
  26. 2020ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unitcited by 31
  27. 2018Weakly Supervised Deep Recurrent Neural Networks for Basic Dance Step Generationcited by 22
  28. 2020Temporally Guided Music-to-Body-Movement Generationcited by 21
  29. 2019Generative Autoregressive Networks for 3D Dancing Move Synthesis from Musiccited by 14
  30. 2018Bio-LSTM: A Biomechanically Inspired Recurrent Neural Network for 3D Pedestrian Pose and Gait Predictioncited by 13
  31. 2018Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Networkcited by 11
  32. 2020Towards 3D Dance Motion Synthesis and Controlcited by 7
  33. 2019Music-oriented Dance Video Synthesis with Pose Perceptual Losscited by 5
  34. 2020Feel The Music: Automatically Generating A Dance For An Input Songcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 11; 2022: 62; 2023: 100; 2024: 106; 2025: 126; 2026: 74.

479 papers.

Sort:

By area:

  1. 2022Human Motion Diffusion Modelcited by 1,062
  2. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  3. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  4. 2022Executing your Commands via Motion Diffusion in Latent Spacecited by 567
  5. 2021Action-Conditioned 3D Human Motion Synthesis with Transformer VAEcited by 497
  6. 2022TEMOS: Generating diverse human motions from textual descriptionscited by 486
  7. 2022MotionCLIP: Exposing Human Motion Generation to CLIP Spacecited by 406
  8. 2022Multimodal Learning with Transformers: A Surveycited by 399
  9. 2022PhysDiff: Physics-Guided Human Motion Diffusion Modelcited by 365
  10. 2022EDGE: Editable Dance Generation From Musiccited by 284
  11. 2023OmniControl: Control Any Joint at Any Time for Human Motion Generationcited by 233
  12. 2023BEDLAM: A Synthetic Dataset of Bodies Exhibiting Detailed Lifelike Animated Motioncited by 229
38 more
  1. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  2. 2023Guided Motion Diffusion for Controllable Human Motion Synthesiscited by 221
  3. 2023Motion-X: A Large-scale 3D Expressive Whole-body Human Motion Datasetcited by 220
  4. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  5. 2022Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memorycited by 210
  6. 2023InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactionscited by 205
  7. 20233DGS-Avatar: Animatable Avatars via Deformable 3D Gaussian Splattingcited by 195
  8. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  9. 2022BEAT: A Large-Scale Semantic and Emotional Multi-Modal Dataset for Conversational Gestures Synthesiscited by 177
  10. 2022TEACH: Temporal Action Composition for 3D Humanscited by 173
  11. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  12. 2021Neural Human Performer: Learning Generalizable Radiance Fields for Human Performance Renderingcited by 158
  13. 2021GLAMR: Global Occlusion-Aware Human Mesh Recovery with Dynamic Camerascited by 151
  14. 2023MotionGPT: Finetuned LLMs Are General-Purpose Motion Generatorscited by 145
  15. 2021GOAL: Generating 4D Whole-Body Motion for Hand-Object Graspingcited by 142
  16. 2023DisCo: Disentangled Control for Realistic Human Dance Generationcited by 141
  17. 2022ARAH: Animatable Volume Rendering of Articulated Human SDFscited by 139
  18. 2022COUCH: Towards Controllable Human-Chair Interactionscited by 134
  19. 2023Taming Diffusion Models for Audio-Driven Co-Speech Gesture Generationcited by 128
  20. 2023State of the Art on Diffusion Models for Visual Computingcited by 126
  21. 2022HuMMan: Multi-Modal 4D Human Dataset for Versatile Sensing and Modelingcited by 115
  22. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  23. 2022Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videoscited by 110
  24. 2023Avatars Grow Legs: Generating Smooth Human Motion from Sparse Tracking Inputs with Diffusion Modelcited by 104
  25. 2023DNA-Rendering: A Diverse Neural Actor Repository for High-Fidelity Human-centric Renderingcited by 103
  26. 2022Generative Neural Articulated Radiance Fieldscited by 101
  27. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  28. 2023Inter-X: Towards Versatile Human-Human Interaction Analysiscited by 98
  29. 2023Optimizing Diffusion Noise Can Serve As Universal Motion Priorscited by 97
  30. 2023Human Motion Generation: A Surveycited by 95
  31. 2023TLControl: Trajectory and Language Control for Human Motion Synthesiscited by 93
  32. 2023EMDM: Efficient Motion Diffusion Model for Fast and High-Quality Motion Generationcited by 88
  33. 2024Multi-Track Timeline Control for Text-Driven 3D Human Motion Generationcited by 88
  34. 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 87
  35. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  36. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 83
  37. 2022GIMO: Gaze-Informed Human Motion Prediction in Contextcited by 81
  38. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81

50 most cited of 479 All 479 (JSON)