# Explore 3D Dance Generation via Reward Model from Automatically-Ranked Demonstrations

arXiv: https://arxiv.org/abs/2312.11442 · HTML: https://mlsanity.com/paper/2312.11442/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 66 references that are in the map.
- First posted on arXiv 18 Dec 2023.

## Builds on (most cited first, 30 of 33)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Reward learning from human preferences and demonstrations in Atari](https://mlsanity.com/paper/1811.06521/) (2018)
- [Teaching language models to support answers with verified quotes](https://mlsanity.com/paper/2203.11147/) (2022)
- [Recursively Summarizing Books with Human Feedback](https://mlsanity.com/paper/2109.10862/) (2021)
- [Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations](https://mlsanity.com/paper/1904.06387/) (2019)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [Interactive Learning from Policy-Dependent Human Feedback](https://mlsanity.com/paper/1701.06049/) (2017)
- [Prompting Decision Transformer for Few-Shot Policy Generalization](https://mlsanity.com/paper/2206.13499/) (2022)
- [Deep TAMER: Interactive Agent Shaping in High-Dimensional State Spaces](https://mlsanity.com/paper/1709.10163/) (2017)
- [DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer](https://mlsanity.com/paper/2103.10206/) (2021)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [Few-Shot Preference Learning for Human-in-the-Loop RL](https://mlsanity.com/paper/2212.03363/) (2022)
- [Learning to Generate Diverse Dance Motions with Transformer](https://mlsanity.com/paper/2008.08171/) (2020)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [Preference Transformer: Modeling Human Preferences using Transformers for RL](https://mlsanity.com/paper/2303.00957/) (2023)
- [Learning to Paint With Model-based Deep Reinforcement Learning](https://mlsanity.com/paper/1903.04411/) (2019)
- [Combating the Compounding-Error Problem with a Multi-step Model](https://mlsanity.com/paper/1905.13320/) (2019)
- [ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit](https://mlsanity.com/paper/2009.07637/) (2020)
- [Inverse reinforcement learning for video games](https://mlsanity.com/paper/1810.10593/) (2018)

## Built on by (most cited first, 8 of 8)

- [Co-Speech Gesture Video Generation via Motion-Decoupled Diffusion Model](https://mlsanity.com/paper/2404.01862/) (2024)
- [Exploring Text-to-Motion Generation with Human Preference](https://mlsanity.com/paper/2404.09445/) (2024)
- [Align Your Rhythm: Generating Highly Aligned Dance Poses with Gating-Enhanced Rhythm-Aware Feature Representation](https://mlsanity.com/paper/2503.17340/) (2025)
- [Tempo as the Stable Cue: Hierarchical Mixture of Tempo and Beat Experts for Music to 3D Dance Generation](https://mlsanity.com/paper/2512.18804/) (2025)
- [EchoAvatar: Real-time Generative Avatar Animation from Audio Streams](https://mlsanity.com/paper/2605.28272/) (2026)
- [Continuous Vision-Language-Action Co-Learning with Semantic-Physical Alignment for Behavioral Cloning](https://mlsanity.com/paper/2511.14396/) (2025)
- [FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams](https://mlsanity.com/paper/2608.15818/) (2026)
- [BeatDance: Generating Beat-Consistent 3D Dance with Hierarchical Spatial-Temporal Modeling](https://mlsanity.com/paper/2609.37400/) (2026)

## Abstract (arXiv)

This paper presents an Exploratory 3D Dance generation framework, E3D2, designed to address the exploration capability deficiency in existing music-conditioned 3D dance generation models. Current models often generate monotonous and simplistic dance sequences that misalign with human preferences because they lack exploration capabilities. The E3D2 framework involves a reward model trained from automatically-ranked dance demonstrations, which then guides the reinforcement learning process. This approach encourages the agent to explore and generate high quality and diverse dance movement sequences. The soundness of the reward model is both theoretically and experimentally validated. Empirical experiments demonstrate the effectiveness of E3D2 on the AIST++ dataset. Project Page: https://sites.google.com/view/e3d2.