mlsanityData to 7 Oct 2026

Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory

arXiv 2203.13055 ↗First posted 24 Mar 2022cs.SD cs.CV eess.AScited by 210

Not on the map. Why?

Cited by 32 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Driving 3D characters to dance following a piece of music is highly challenging due to the spatial constraints applied to poses by choreography norms. In addition, the generated dance sequence also needs to maintain temporal coherency with different music genres. To tackle these challenges, we propose a novel music-to-dance framework, Bailando, with two powerful components: 1) a choreographic memory that learns to summarize meaningful dancing units from 3D pose sequence to a quantized codebook, 2) an actor-critic Generative Pre-trained Transformer (GPT) that composes these units to a fluent dance coherent to the music. With the learned choreographic memory, dance generation is realized on the quantized units that meet high choreography standards, such that the generated dancing sequences are confined within the spatial constraints. To achieve synchronized alignment between diverse motion tempos and music beats, we introduce an actor-critic-based reinforcement learning scheme to the GPT with a newly-designed beat-align reward function. Extensive experiments on the standard benchmark demonstrate that our proposed framework achieves state-of-the-art performance both qualitatively and quantitatively. Notably, the learned choreographic memory is shown to discover human-interpretable dancing-style poses in an unsupervised manner.

Key facts. Cited by 210 papers in the map (data through 7 Oct 2026). Builds on 19 of its 42 references that are in the map. First posted on arXiv 24 Mar 2022.

Builds on

19 of 42 references.

References · 19

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Neural Discrete Representation Learningcited by 4,650
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  7. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 651
  8. 2020Jukebox: A Generative Model for Musiccited by 535
  9. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  10. 2019Learning Individual Styles of Conversational Gesturecited by 190
  11. 2021DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformercited by 111
  12. 2020Learning to Generate Diverse Dance Motions with Transformercited by 85
7 more
  1. 2020Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learningcited by 81
  2. 2020Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audiocited by 33
  3. 2020ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unitcited by 31
  4. 2017Towards the Automatic Anime Characters Creation with Generative Adversarial Networkscited by 29
  5. 2018Weakly Supervised Deep Recurrent Neural Networks for Basic Dance Step Generationcited by 22
  6. 2020Temporally Guided Music-to-Body-Movement Generationcited by 21
  7. 2019Generative Autoregressive Networks for 3D Dancing Move Synthesis from Musiccited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 13; 2023: 43; 2024: 49; 2025: 61; 2026: 44.

210 papers.

Sort:

By area:

  1. 2022MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Modelcited by 768
  2. 2023T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representationscited by 577
  3. 2023MotionGPT: Human Motion as a Foreign Languagecited by 521
  4. 2023MoMask: Generative Masked Modeling of 3D Human Motionscited by 388
  5. 2022SadTalker: Learning Realistic 3D Motion Coefficients for Stylized Audio-Driven Single Image Talking Face Animationcited by 380
  6. 2022EDGE: Editable Dance Generation From Musiccited by 284
  7. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  8. 2022Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Modelscited by 188
  9. 2024VASA-1: Lifelike Audio-Driven Talking Faces Generated in Real Timecited by 186
  10. 2022Generating Holistic 3D Human Motion from Speechcited by 180
  11. 2024MotionLCM: Real-time Controllable Motion Generation via Latent Consistency Modelcited by 164
  12. 2023HumanTOMATO: Text-aligned Whole-body Motion Generationcited by 103
38 more
  1. 2023Human Motion Generation: A Surveycited by 95
  2. 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 87
  3. 2023DiffuseStyleGesture: Stylized Audio-Driven Co-Speech Gesture Generation with Diffusion Modelscited by 85
  4. 2025SONIC: Supersizing Motion Tracking for Natural Humanoid Whole-Body Controlcited by 83
  5. 2023HumanMAC: Masked Motion Completion for Human Motion Predictioncited by 81
  6. 2023DiffPoseTalk: Speech-Driven Stylistic 3D Facial Animation and Head Pose Generation via Diffusion Modelscited by 81
  7. 2024DartControl: A Diffusion-Based Autoregressive Motion Model for Real-Time Text-Driven Motion Controlcited by 80
  8. 2022PoseGPT: Quantization-based 3D Human Motion Generation and Forecastingcited by 78
  9. 2023TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integrationcited by 76
  10. 2022UDE: A Unified Driving Engine for Human Motion Generationcited by 67
  11. 2024CoMo: Controllable Motion Generation through Language Guided Pose Code Editingcited by 66
  12. 2024Large Motion Model for Unified Multi-Modal Motion Generationcited by 65
  13. 2024Lodge: A Coarse to Fine Diffusion Network for Long Dance Generation Guided by the Characteristic Dance Primitivescited by 59
  14. 2023MoConVQ: Unified Physics-Based Motion Control via Scalable Discrete Representationscited by 58
  15. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 54
  16. 2023Text2Performer: Text-Driven Human Video Generationcited by 53
  17. 2024ParCo: Part-Coordinating Text-to-Motion Synthesiscited by 51
  18. 2024MambaTalk: Efficient Holistic Gesture Synthesis with Selective State Space Modelscited by 46
  19. 2025GENMO: A GENeralist Model for Human MOtioncited by 43
  20. 2024ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesiscited by 41
  21. 2023QPGesture: Quantization-Based and Phase-Guided Motion Matching for Natural Speech-Driven Gesture Generationcited by 40
  22. 2023LivelySpeaker: Towards Semantic-Aware Co-Speech Gesture Generationcited by 40
  23. 2025Motion Anything: Any to Motion Generationcited by 40
  24. 2022Rhythmic Gesticulator: Rhythm-Aware Co-Speech Gesture Synthesis with Hierarchical Neural Embeddingscited by 38
  25. 2024M3GPT: An Advanced Multimodal, Multitask Framework for Motion Comprehension and Generationcited by 37
  26. 2023Music-Driven Group Choreographycited by 36
  27. 2025MotionGPT3: Human Motion as a Second Modalitycited by 36
  28. 2024Assessing and Understanding Creativity in Large Language Modelscited by 35
  29. 2024Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Modelcited by 34
  30. 2024MotionChain: Conversational Motion Controllers via Multimodal Promptscited by 30
  31. 2024Towards Variable and Coordinated Holistic Co-Speech Motion Generationcited by 29
  32. 2022Pretrained Diffusion Models for Unified Human Motion Synthesiscited by 26
  33. 2025A Survey on Human Interaction Motion Generationcited by 25
  34. 2024MotionCraft: Crafting Whole-Body Motion with Plug-and-Play Multimodal Controlscited by 24
  35. 2023DiffDance: Cascaded Human Motion Diffusion Model for Dance Generationcited by 23
  36. 2024Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patternscited by 21
  37. 2025DuetGen: Music Driven Two-Person Dance Generation via Hierarchical Masked Modelingcited by 21
  38. 2023Dance with You: The Diversity Controllable Dancer Generation via Diffusion Modelscited by 17

50 most cited of 210 All 210 (JSON)