# How Animals Dance (When You're Not Looking)

arXiv: https://arxiv.org/abs/2505.23738 · HTML: https://mlsanity.com/paper/2505.23738/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 36 references that are in the map.
- First posted on arXiv 29 May 2025.

## Builds on (most cited first, 24 of 24)

- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Adding Conditional Control to Text-to-Image Diffusion Models](https://mlsanity.com/paper/2302.05543/) (2023)
- [Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets](https://mlsanity.com/paper/2311.15127/) (2023)
- [RAFT: Recurrent All-Pairs Field Transforms for Optical Flow](https://mlsanity.com/paper/2003.12039/) (2020)
- [CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer](https://mlsanity.com/paper/2408.06072/) (2024)
- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Animate Anyone: Consistent and Controllable Image-to-Video Synthesis for Character Animation](https://mlsanity.com/paper/2311.17117/) (2023)
- [MotionCtrl: A Unified and Flexible Motion Controller for Video Generation](https://mlsanity.com/paper/2312.03641/) (2023)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [Lumiere: A Space-Time Diffusion Model for Video Generation](https://mlsanity.com/paper/2401.12945/) (2024)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation](https://mlsanity.com/paper/2212.09478/) (2022)
- [Motion Prompting: Controlling Video Generation with Motion Trajectories](https://mlsanity.com/paper/2412.02700/) (2024)
- [Make Pixels Dance: High-Dynamic Video Generation](https://mlsanity.com/paper/2311.10982/) (2023)
- [World-Grounded Human Motion Recovery via Gravity-View Coordinates](https://mlsanity.com/paper/2409.06662/) (2024)
- [Animate-X: Universal Character Image Animation with Enhanced Motion Representation](https://mlsanity.com/paper/2410.10306/) (2024)
- [Dance Revolution: Long-Term Dance Generation with Music via Curriculum Learning](https://mlsanity.com/paper/2006.06119/) (2020)
- [Animate Anyone 2: High-Fidelity Character Image Animation with Environment Affordance](https://mlsanity.com/paper/2502.06145/) (2025)
- [Generative Inbetweening: Adapting Image-to-Video Models for Keyframe Interpolation](https://mlsanity.com/paper/2408.15239/) (2024)
- [Framer: Interactive Frame Interpolation](https://mlsanity.com/paper/2410.18978/) (2024)
- [DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation](https://mlsanity.com/paper/2308.02915/) (2023)
- [Controllable Group Choreography using Contrastive Diffusion](https://mlsanity.com/paper/2310.18986/) (2023)
- [Listen to Dance: Music-driven choreography generation using Autoregressive Encoder-Decoder Network](https://mlsanity.com/paper/1811.00818/) (2018)
- [MusicInfuser: Making Video Diffusion Listen and Dance](https://mlsanity.com/paper/2503.14505/) (2025)

## Abstract (arXiv)

We present a framework for generating music-synchronized, choreography aware animal dance videos. Our framework introduces choreography patterns -- structured sequences of motion beats that define the long-range structure of a dance -- as a novel high-level control signal for dance video generation. These patterns can be automatically estimated from human dance videos. Starting from a few keyframes representing distinct animal poses, generated via text-to-image prompting or GPT-4o, we formulate dance synthesis as a graph optimization problem that seeks the optimal keyframe structure to satisfy a specified choreography pattern of beats. We also introduce an approach for mirrored pose image generation, essential for capturing symmetry in dance. In-between frames are synthesized using an video diffusion model. With as few as six input keyframes, our method can produce up to 30 seconds dance videos across a wide range of animals and music tracks.