mlsanityData to 7 Oct 2026

Audio to Body Dynamics

arXiv 1712.09382 ↗First posted 19 Dec 2017eess.AS cs.CV cs.SDcited by 48

Not on the map. Why?

Abstract (arXiv)

We present a method that gets as input an audio of violin or piano playing, and outputs a video of skeleton predictions which are further used to animate an avatar. The key idea is to create an animation of an avatar that moves their hands similarly to how a pianist or violinist would do, just from audio. Aiming for a fully detailed correct arms and fingers motion is a goal, however, it's not clear if body movement can be predicted from music at all. In this paper, we present the first result that shows that natural body dynamics can be predicted at all. We built an LSTM network that is trained on violin and piano recital videos uploaded to the Internet. The predicted points are applied onto a rigged avatar to create the animation.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). First posted on arXiv 19 Dec 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 12; 2023: 8; 2024: 8; 2025: 5; 2026: 5.

48 papers.

Sort:

By area:

  1. 2021AI Choreographer: Music Conditioned 3D Dance Generation with AIST++cited by 479
  2. 2022MoFusion: A Framework for Denoising-Diffusion-based Motion Synthesiscited by 218
  3. 2021Audio2Gestures: Generating Diverse Gestures from Speech Audio with Conditional Variational Autoencoderscited by 109
  4. 2021Learning Speech-driven 3D Conversational Gestures from Videocited by 98
  5. 2021Speech Drives Templates: Co-Speech Gesture Synthesis with Learned Templatescited by 57
  6. 2021Rhythm is a Dancer: Music-Driven Motion Synthesis with Global Structurecited by 53
  7. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  8. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  9. 2023Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answeringcited by 28
  10. 2022Weakly-supervised Action Transition Learning for Stochastic Human Motion Predictioncited by 26
  11. 2022FLAG3D: A 3D Fitness Activity Dataset with Language Instructioncited by 23
  12. 2023DiffDance: Cascaded Human Motion Diffusion Model for Dance Generationcited by 23
36 more
  1. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  2. 2023BodyFormer: Semantics-guided 3D Body Gesture Synthesis with Transformercited by 21
  3. 2021Action2video: Generating Videos of Human 3D Actionscited by 15
  4. 2024Dance Any Beat: Blending Beats with Visuals in Dance Video Generationcited by 14
  5. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 14
  6. 2022Finding Fallen Objects Via Asynchronous Audio-Visual Integrationcited by 13
  7. 2021FixMyPose: Pose Correctional Captioning and Retrievalcited by 12
  8. 2022Freeform Body Motion Generation from Speechcited by 11
  9. 2021Egocentric Videoconferencingcited by 10
  10. 2023Audio2Gestures: Generating Diverse Gestures from Audiocited by 10
  11. 2023From Pixels to Portraits: A Comprehensive Survey of Talking Head Generation Techniques and Applicationscited by 9
  12. 2022PaCMO: Partner Dependent Human Motion Generation in Dyadic Human Activity using Neural Operatorscited by 8
  13. 2024F"urElise: Capturing and Physically Synthesizing Hand Motions of Piano Performancecited by 8
  14. 2022Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encodingcited by 7
  15. 2021TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separationcited by 6
  16. 2022Programmatic Concept Learning for Human Motion Description and Synthesiscited by 5
  17. 2023Spatial-temporal Transformer-guided Diffusion based Data Augmentation for Efficient Skeleton-based Action Recognitioncited by 4
  18. 2025Spatial-Temporal Graph Mamba for Music-Guided Dance Video Synthesiscited by 4
  19. 2025ChoreoMuse: Robust Music-to-Dance Video Generation with Style Transfer and Beat-Adherent Motioncited by 4
  20. 2022AIMusicGuru: Music Assisted Human Pose Correctioncited by 3
  21. 2024MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processingcited by 3
  22. 2024Acoustic-based 3D Human Pose Estimation Robust to Human Positioncited by 2
  23. 2026SignSparK: Efficient Multilingual Sign Language Production via Sparse Keyframe Learningcited by 2
  24. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  25. 2023Cross-modal Generative Model for Visual-Guided Binaural Stereo Generationcited by 1
  26. 2024CLHOP: Combined Audio-Video Learning for Horse 3D Pose and Shape Estimationcited by 1
  27. 2024STNet: Deep Audio-Visual Fusion Network for Robust Speaker Trackingcited by 1
  28. 2025BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Soundscited by 1
  29. 2026MUSIC: Learning Muscle-Driven Dexterous Hand Controlcited by 1
  30. 2021Multi-Modulation Network for Audio-Visual Event Localizationcited by 0
  31. 2023ZS-MSTM: Zero-Shot Style Transfer for Gesture Animation driven by Text and Speech using Adversarial Disentanglement of Multimodal Style Encodingcited by 0
  32. 2024VioPose: Violin Performance 4D Pose Estimation by Hierarchical Audiovisual Inferencecited by 0
  33. 2025Audio-3DVG: Unified Audio -- Point Cloud Fusion for 3D Visual Groundingcited by 0
  34. 2025Stochastic Human Motion Prediction with Memory of Action Transition and Action Characteristiccited by 0
  35. 2026Tipiano: Cascaded Piano Hand Motion Synthesis via Fingertip Priorscited by 0
  36. 2026Super Star: Towards Streaming Real-time Interactive Agents for Digital Humanscited by 0