mlsanityData to 7 Oct 2026

Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audio

arXiv 2311.06285 ↗First posted 1 Nov 2023cs.CV cs.LG cs.SDcited by 6

Not on the map. Why?

Abstract (arXiv)

While 3D human body modeling has received much attention in computer vision, modeling the acoustic equivalent, i.e. modeling 3D spatial audio produced by body motion and speech, has fallen short in the community. To close this gap, we present a model that can generate accurate 3D spatial audio for full human bodies. The system consumes, as input, audio signals from headset microphones and body pose, and produces, as output, a 3D sound field surrounding the transmitter's body, from which spatial audio can be rendered at any arbitrary position in the 3D space. We collect a first-of-its-kind multimodal dataset of human bodies, recorded with multiple cameras and a spherical array of 345 microphones. In an empirical evaluation, we demonstrate that our model can produce accurate body-induced sound fields when trained with a suitable loss. Dataset and code are available online.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 39 of its 52 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

39 of 52 references.

References · 39

  1. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  2. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  3. 2017End-to-end Recovery of Human Shape and Posecited by 883
  4. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  5. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  6. 2018The Sound of Pixelscited by 258
  7. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  8. 2020Speech Gesture Generation from the Trimodal Context of Text, Audio, and Speaker Identitycited by 199
  9. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  10. 2019Learning Individual Styles of Conversational Gesturecited by 190
  11. 2018Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separationcited by 160
  12. 2020Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsingcited by 146
27 more
  1. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  2. 2020Multiple Sound Sources Localization from Coarse to Finecited by 118
  3. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  4. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  5. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  6. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  7. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  8. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  9. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  10. 2023AvatarReX: Real-time Expressive Full-body Avatarscited by 83
  11. 2021Driving-Signal Aware Full-Body Avatarscited by 81
  12. 2022Localizing Visual Sounds the Easy Waycited by 81
  13. 2021Robust Audio-Visual Instance Discriminationcited by 80
  14. 2016Ambient Sound Provides Supervision for Visual Learningcited by 77
  15. 2020Style Transfer for Co-Speech Gesture Animation: A Multi-Speaker Conditional-Mixture Approachcited by 64
  16. 2021Cyclic Co-Learning of Sounding Object Visual Grounding and Sound Separationcited by 59
  17. 2020Sep-Stereo: Visually Guided Stereophonic Audio Generation by Associating Source Separationcited by 55
  18. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  19. 2020Telling Left from Right: Learning Spatial Correspondence of Sight and Soundcited by 43
  20. 2022Egocentric Deep Multi-Channel Audio-Visual Active Speaker Localizationcited by 27
  21. 2018Scene-Aware Audio for 360textdegree Videoscited by 25
  22. 2022Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesiscited by 24
  23. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  24. 2022Deep Impulse Responses: Estimating and Parameterizing Filters with Deep Networkscited by 20
  25. 2019Self-supervised Audio Spatialization with Correspondence Classifiercited by 10
  26. 2020Parallel waveform synthesis based on generative adversarial networks with voicing-aware conditional discriminatorscited by 7
  27. 2020Noisy Agents: Self-supervised Exploration by Predicting Auditory Eventscited by 5

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 4.

6 papers.

  1. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  2. 2025MOSPA: Human Motion Generation Driven by Spatial Audiocited by 11
  3. 2025Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenescited by 2
  4. 2025BGM2Pose: Active 3D Human Pose Estimation with Non-Stationary Soundscited by 1
  5. 2025In-the-wild Audio Spatialization with Flexible Text-guided Localizationcited by 1
  6. 2024Zero-Shot Mono-to-Binaural Speech Synthesiscited by 0