# VirtualConductor: Music-driven Conducting Video Generation System

arXiv: https://arxiv.org/abs/2108.04350 · HTML: https://mlsanity.com/paper/2108.04350/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 4 references that are in the map.
- First posted on arXiv 28 Jul 2021.

## Builds on (most cited first, 2 of 2)

- [RMPE: Regional Multi-person Pose Estimation](https://mlsanity.com/paper/1612.00137/) (2016)
- [Liquid Warping GAN: A Unified Framework for Human Motion Imitation, Appearance Transfer and Novel View Synthesis](https://mlsanity.com/paper/1909.12224/) (2019)

## Built on by (most cited first, 1 of 1)

- [Taming Diffusion Models for Music-driven Conducting Motion Generation](https://mlsanity.com/paper/2306.10065/) (2023)

## Abstract (arXiv)

In this demo, we present VirtualConductor, a system that can generate conducting video from any given music and a single user's image. First, a large-scale conductor motion dataset is collected and constructed. Then, we propose Audio Motion Correspondence Network (AMCNet) and adversarial-perceptual learning to learn the cross-modal relationship and generate diverse, plausible, music-synchronized motion. Finally, we combine 3D animation rendering and a pose transfer model to synthesize conducting video from a single given user's image. Therefore, any user can become a virtual conductor through the system.