# Dance Any Beat: Blending Beats with Visuals in Dance Video Generation

arXiv: https://arxiv.org/abs/2405.09266 · HTML: https://mlsanity.com/paper/2405.09266/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 55 of its 73 references that are in the map.
- First posted on arXiv 15 May 2024.

## Builds on (most cited first, 30 of 55)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [The Unreasonable Effectiveness of Deep Features as a Perceptual Metric](https://mlsanity.com/paper/1801.03924/) (2018)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [You Only Look Once: Unified, Real-Time Object Detection](https://mlsanity.com/paper/1506.02640/) (2015)
- [Improved Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2102.09672/) (2021)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [Video Diffusion Models](https://mlsanity.com/paper/2204.03458/) (2022)
- [3D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotation](https://mlsanity.com/paper/1606.06650/) (2016)
- [Spatial Transformer Networks](https://mlsanity.com/paper/1506.02025/) (2015)
- [Towards Accurate Generative Models of Video: A New Metric & Challenges](https://mlsanity.com/paper/1812.01717/) (2018)
- [Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields](https://mlsanity.com/paper/1611.08050/) (2016)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [First Order Motion Model for Image Animation](https://mlsanity.com/paper/2003.00196/) (2020)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Video-to-Video Synthesis](https://mlsanity.com/paper/1808.06601/) (2018)
- [Everybody Dance Now](https://mlsanity.com/paper/1808.07371/) (2018)
- [Few-Shot Adversarial Learning of Realistic Neural Talking Head Models](https://mlsanity.com/paper/1905.08233/) (2019)
- [Motion Representations for Articulated Animation](https://mlsanity.com/paper/2104.11280/) (2021)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Wav2CLIP: Learning Robust Audio Representations From CLIP](https://mlsanity.com/paper/2110.11499/) (2021)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [Thin-Plate Spline Motion Model for Image Animation](https://mlsanity.com/paper/2203.14367/) (2022)
- [Few-shot Video-to-Video Synthesis](https://mlsanity.com/paper/1910.12713/) (2019)
- [Learning Individual Styles of Conversational Gesture](https://mlsanity.com/paper/1906.04160/) (2019)
- [X2Face: A network for controlling face generation by using images, audio, and pose codes](https://mlsanity.com/paper/1807.10550/) (2018)
- [Conditional Image-to-Video Generation with Latent Flow Diffusion Models](https://mlsanity.com/paper/2303.13744/) (2023)
- [Latent Image Animator: Learning to Animate Images via Latent Space Navigation](https://mlsanity.com/paper/2203.09043/) (2022)

## Built on by (most cited first, 14 of 14)

- [OpenHumanVid: A Large-Scale High-Quality Dataset for Enhancing Human-Centric Video Generation](https://mlsanity.com/paper/2412.00115/) (2024)
- [A Comprehensive Survey on Human Video Generation: Challenges, Methods, and Insights](https://mlsanity.com/paper/2407.08428/) (2024)
- [Human Motion Video Generation: A Survey](https://mlsanity.com/paper/2509.03883/) (2025)
- [X-Dancer: Expressive Music to Human Dance Video Generation](https://mlsanity.com/paper/2502.17414/) (2025)
- [Audio-Sync Video Generation with Multi-Stream Temporal Control](https://mlsanity.com/paper/2506.08003/) (2025)
- [MACE-Dance: Motion-Appearance Cascaded Experts for Music-Driven Dance Video Generation](https://mlsanity.com/paper/2512.18181/) (2025)
- [Video Motion Graphs](https://mlsanity.com/paper/2503.20218/) (2025)
- [ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motion](https://mlsanity.com/paper/2507.19836/) (2025)
- [OmniDance: Multimodal Driven Dance Video Generation with Large-scale Internet Data](https://mlsanity.com/paper/2606.30019/) (2026)
- [MotionBeat: Motion-Aligned Music Representation via Embodied Contrastive Learning and Bar-Equivariant Contact-Aware Encoding](https://mlsanity.com/paper/2510.13244/) (2025)
- [Generative Augmented Reality: Paradigms, Technologies, and Future Applications](https://mlsanity.com/paper/2511.16783/) (2025)
- [Wan-Dancer: A Hierarchical Framework for Minute-scale Coherent Music-to-Dance Generation](https://mlsanity.com/paper/2607.09581/) (2026)
- [Learning Quantised Structure-Preserving Motion Representations for Dance Fingerprinting](https://mlsanity.com/paper/2604.00927/) (2026)
- [FlowDance: Music-Driven Dance Video Generation with Parallel Pose and RGB Streams](https://mlsanity.com/paper/2608.15818/) (2026)

## Abstract (arXiv)

Generating dance from music is crucial for advancing automated choreography. Current methods typically produce skeleton keypoint sequences instead of dance videos and lack the capability to make specific individuals dance, which reduces their real-world applicability. These methods also require precise keypoint annotations, complicating data collection and limiting the use of self-collected video datasets. To overcome these challenges, we introduce a novel task: generating dance videos directly from images of individuals guided by music. This task enables the dance generation of specific individuals without requiring keypoint annotations, making it more versatile and applicable to various situations. Our solution, the Dance Any Beat Diffusion model (DabFusion), utilizes a reference image and a music piece to generate dance videos featuring various dance types and choreographies. The music is analyzed by our specially designed music encoder, which identifies essential features including dance style, movement, and rhythm. DabFusion excels in generating dance videos not only for individuals in the training dataset but also for any previously unseen person. This versatility stems from its approach of generating latent optical flow, which contains all necessary motion information to animate any person in the image. We evaluate DabFusion's performance using the AIST++ dataset, focusing on video quality, audio-video synchronization, and motion-music alignment. We propose a 2D Motion-Music Alignment Score (2D-MM Align), which builds on the Beat Alignment Score to more effectively evaluate motion-music alignment for this new task. Experiments show that our DabFusion establishes a solid baseline for this innovative task. Video results can be found on our project page: https://DabFusion.github.io.