# MDD: A Dataset for Text-and-Music Conditioned Duet Dance Generation

arXiv: https://arxiv.org/abs/2508.16911 · HTML: https://mlsanity.com/paper/2508.16911/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 51 references that are in the map.
- First posted on arXiv 23 Aug 2025.

## Builds on (most cited first, 30 of 33)

- [Expressive Body Capture: 3D Hands, Face, and Body from a Single Image](https://mlsanity.com/paper/1904.05866/) (2019)
- [Human Motion Diffusion Model](https://mlsanity.com/paper/2209.14916/) (2022)
- [MotionDiffuse: Text-Driven Human Motion Generation with Diffusion Model](https://mlsanity.com/paper/2208.15001/) (2022)
- [T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations](https://mlsanity.com/paper/2301.06052/) (2023)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [TEMOS: Generating diverse human motions from textual descriptions](https://mlsanity.com/paper/2204.14109/) (2022)
- [AI Choreographer: Music Conditioned 3D Dance Generation with AIST++](https://mlsanity.com/paper/2101.08779/) (2021)
- [MotionCLIP: Exposing Human Motion Generation to CLIP Space](https://mlsanity.com/paper/2203.08063/) (2022)
- [EDGE: Editable Dance Generation From Music](https://mlsanity.com/paper/2211.10658/) (2022)
- [Bailando: 3D Dance Generation by Actor-Critic GPT with Choreographic Memory](https://mlsanity.com/paper/2203.13055/) (2022)
- [InterGen: Diffusion-based Multi-human Motion Generation under Complex Interactions](https://mlsanity.com/paper/2304.05684/) (2023)
- [Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models](https://mlsanity.com/paper/2211.09707/) (2022)
- [DisCo: Disentangled Control for Realistic Human Dance Generation](https://mlsanity.com/paper/2307.00040/) (2023)
- [Text2Gestures: A Transformer-Based Network for Generating Emotive Body Gestures for Virtual Agents](https://mlsanity.com/paper/2101.11101/) (2021)
- [Inter-X: Towards Versatile Human-Human Interaction Analysis](https://mlsanity.com/paper/2312.16051/) (2023)
- [FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation](https://mlsanity.com/paper/2212.03741/) (2022)
- [Rethinking Diffusion for Text-Driven Human Motion Generation: Redundant Representations, Evaluation, and Masked Autoregression](https://mlsanity.com/paper/2411.16575/) (2024)
- [Generating Human Interaction Motions in Scenes with Text Control](https://mlsanity.com/paper/2404.10685/) (2024)
- [Fg-T2M: Fine-Grained Text-Driven Human Motion Generation via Diffusion Model](https://mlsanity.com/paper/2309.06284/) (2023)
- [TM2D: Bimodality Driven 3D Dance Generation via Music-Text Integration](https://mlsanity.com/paper/2304.02419/) (2023)
- [MotionFix: Text-Driven 3D Human Motion Editing](https://mlsanity.com/paper/2408.00712/) (2024)
- [ReMoS: 3D Motion-Conditioned Reaction Synthesis for Two-Person Interactions](https://mlsanity.com/paper/2311.17057/) (2023)
- [ReGenNet: Towards Human Action-Reaction Synthesis](https://mlsanity.com/paper/2403.11882/) (2024)
- [OMG: Towards Open-vocabulary Motion Generation via Mixture of Controllers](https://mlsanity.com/paper/2312.08985/) (2023)
- [Duolando: Follower GPT with Off-Policy Reinforcement Learning for Dance Accompaniment](https://mlsanity.com/paper/2403.18811/) (2024)
- [Music-Driven Group Choreography](https://mlsanity.com/paper/2303.12337/) (2023)
- [Ready-to-React: Online Reaction Policy for Two-Character Interaction Generation](https://mlsanity.com/paper/2502.20370/) (2025)
- [SimMotionEdit: Text-Based Human Motion Editing with Motion Similarity Prediction](https://mlsanity.com/paper/2503.18211/) (2025)
- [POPDG: Popular 3D Dance Generation with PopDanceSet](https://mlsanity.com/paper/2405.03178/) (2024)
- [InterDance:Reactive 3D Dance Generation with Realistic Duet Interactions](https://mlsanity.com/paper/2412.16982/) (2024)

## Built on by (most cited first, 4 of 4)

- [Unified Multi-Modal Interactive & Reactive 3D Motion Generation via Rectified Flow](https://mlsanity.com/paper/2509.24099/) (2025)
- [Sketch2Colab: Sketch-Conditioned Multi-Human Animation via Controllable Flow Distillation](https://mlsanity.com/paper/2603.02190/) (2026)
- [Text Dictates, Music Decorates: Energy-based Attention for Editable Dance Motion Generation](https://mlsanity.com/paper/2606.22726/) (2026)
- [MIME: Multimodal Interactive Motion Encoder](https://mlsanity.com/paper/2607.22702/) (2026)

## Abstract (arXiv)

We introduce Multimodal DuetDance (MDD), a diverse multimodal benchmark dataset designed for text-controlled and music-conditioned 3D duet dance motion generation. Our dataset comprises 620 minutes of high-quality motion capture data performed by professional dancers, synchronized with music, and detailed with over 10K fine-grained natural language descriptions. The annotations capture a rich movement vocabulary, detailing spatial relationships, body movements, and rhythm, making MDD the first dataset to seamlessly integrate human motions, music, and text for duet dance generation. We introduce two novel tasks supported by our dataset: (1) Text-to-Duet, where given music and a textual prompt, both the leader and follower dance motion are generated (2) Text-to-Dance Accompaniment, where given music, textual prompt, and the leader's motion, the follower's motion is generated in a cohesive, text-aligned manner. We include baseline evaluations on both tasks to support future research.