# DanceCamera3D: 3D Camera Movement Synthesis with Music and Dance

arXiv: https://arxiv.org/abs/2403.13667 · HTML: https://mlsanity.com/paper/2403.13667/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 51 references that are in the map.
- First posted on arXiv 20 Mar 2024.

## Builds on (most cited first, 21 of 21)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1812.08008/) (2018)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [AlphaPose: Whole-Body Regional Multi-Person Pose Estimation and Tracking in Real-Time](https://mlsanity.com/paper/2211.03375/) (2022)
- [Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models](https://mlsanity.com/paper/2208.06677/) (2022)
- [DanceFormer: Music Conditioned 3D Dance Generation with Parametric Motion Transformer](https://mlsanity.com/paper/2103.10206/) (2021)
- [FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation](https://mlsanity.com/paper/2212.03741/) (2022)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [A Unified Framework for Shot Type Classification Based on Subject Centric Lens](https://mlsanity.com/paper/2008.03548/) (2020)
- [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023)
- [ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit](https://mlsanity.com/paper/2009.07637/) (2020)
- [Dynamic Storyboard Generation in an Engine-based Virtual Environment for Video Production](https://mlsanity.com/paper/2301.12688/) (2023)
- [Dance with You: The Diversity Controllable Dancer Generation via Diffusion Models](https://mlsanity.com/paper/2308.13551/) (2023)
- [Can a Robot Become a Movie Director? Learning Artistic Principles for Aerial Cinematography](https://mlsanity.com/paper/1904.02579/) (2019)
- [Batteries, camera, action! Learning a semantic control space for expressive robot cinematography](https://mlsanity.com/paper/2011.10118/) (2020)
- [BRACE: The Breakdancing Competition Dataset for Dance Motion Synthesis](https://mlsanity.com/paper/2207.10120/) (2022)
- [Cine-AI: Generating Video Game Cutscenes in the Style of Human Directors](https://mlsanity.com/paper/2208.05701/) (2022)
- [Dual Learning Music Composition and Dance Choreography](https://mlsanity.com/paper/2201.11999/) (2022)

## Built on by (most cited first, 8 of 8)

- [Lodge++: High-quality and Long Dance Generation with Vivid Choreography Patterns](https://mlsanity.com/paper/2410.20389/) (2024)
- [Pulp Motion: Framing-aware multimodal camera and human motion generation](https://mlsanity.com/paper/2510.05097/) (2025)
- [AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning in 4D Scenes](https://mlsanity.com/paper/2510.10670/) (2025)
- [VERTIGO: Visual Preference Optimization for Cinematic Camera Trajectory Generation](https://mlsanity.com/paper/2604.02467/) (2026)
- [Divide and Fuse: Body Part Mesh Recovery from Partially Visible Human Images](https://mlsanity.com/paper/2407.09694/) (2024)
- [InfiniteDance: Scalable 3D Dance Generation Towards in-the-wild Generalization](https://mlsanity.com/paper/2603.13375/) (2026)
- [AESOP: Asymmetric Human-Camera Generation with Translation-Intensity Control](https://mlsanity.com/paper/2609.37229/) (2026)
- [A Diagnostic Study of World-Space Human Recovery Under High Camera Dynamics](https://mlsanity.com/paper/2610.19729/) (2026)

## Abstract (arXiv)

Choreographers determine what the dances look like, while cameramen determine the final presentation of dances. Recently, various methods and datasets have showcased the feasibility of dance synthesis. However, camera movement synthesis with music and dance remains an unsolved challenging problem due to the scarcity of paired data. Thus, we present DCM, a new multi-modal 3D dataset, which for the first time combines camera movement with dance motion and music audio. This dataset encompasses 108 dance sequences (3.2 hours) of paired dance-camera-music data from the anime community, covering 4 music genres. With this dataset, we uncover that dance camera movement is multifaceted and human-centric, and possesses multiple influencing factors, making dance camera synthesis a more challenging task compared to camera or dance synthesis alone. To overcome these difficulties, we propose DanceCamera3D, a transformer-based diffusion model that incorporates a novel body attention loss and a condition separation strategy. For evaluation, we devise new metrics measuring camera movement quality, diversity, and dancer fidelity. Utilizing these metrics, we conduct extensive experiments on our DCM dataset, providing both quantitative and qualitative evidence showcasing the effectiveness of our DanceCamera3D model. Code and video demos are available at https://github.com/Carmenw1203/DanceCamera3D-Official.