mlsanityData to 14 Oct 2026

Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes

arXiv 2506.09989 ↗First posted 11 Jun 2025cs.CVcited by 2

Not on the map. Why?

Abstract (arXiv)

We study the problem of making 3D scene reconstructions interactive by asking the following question: can we predict the sounds of human hands physically interacting with a scene? First, we record a video of a human manipulating objects within a 3D scene using their hands. We then use these action-sound pairs to train a rectified flow model to map 3D hand trajectories to their corresponding audio. At test time, a user can query the model for other actions, parameterized as sequences of hand poses, to estimate their corresponding sounds. In our experiments, we find that our generated sounds accurately convey material properties and actions, and that they are often indistinguishable to human observers from real sounds. Project page: https://www.yimingdou.com/hearing_hands/

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 40 of its 51 references that are in the map. First posted on arXiv 11 Jun 2025.

Builds on

40 of 51 references.

References · 40

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 20233D Gaussian Splatting for Real-Time Radiance Field Renderingcited by 6,226
  4. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,700
  5. 2020D-NeRF: Neural Radiance Fields for Dynamic Scenescited by 1,465
  6. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  7. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  8. 2023Nerfstudio: A Modular Framework for Neural Radiance Field Developmentcited by 600
  9. 2023LERF: Language Embedded Radiance Fieldscited by 543
  10. 2017Look, Listen and Learncited by 458
  11. 2019Learning joint reconstruction of hands and manipulated objectscited by 378
  12. 2023PhysGaussian: Physics-Integrated 3D Gaussians for Generative Dynamicscited by 377
28 more
  1. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  2. 2023Reconstructing Hands in 3D with Transformerscited by 306
  3. 2020Understanding Human Hands in Contact at Internet Scalecited by 281
  4. 2023Multimodal Deep Learningcited by 258
  5. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 251
  6. 2024VR-GS: A Physical Dynamics-Aware Interactive Gaussian Splatting System in Virtual Realitycited by 179
  7. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 144
  8. 2021Taming Visually Guided Sound Generationcited by 136
  9. 2023LLM-Grounder: Open-Vocabulary 3D Visual Grounding with Large Language Model as an Agentcited by 135
  10. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  11. 2015Visually Indicated Soundscited by 128
  12. 2024GARField: Group Anything with Radiance Fieldscited by 109
  13. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  14. 2023Language Embedded Radiance Fields for Zero-Shot Task-Oriented Graspingcited by 91
  15. 2022ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfercited by 75
  16. 2021ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representationscited by 68
  17. 2020Generating Visually Aligned Sound from Videoscited by 67
  18. 2024Robot See Robot Do: Imitating Articulated Object Manipulation with Monocular 4D Reconstructioncited by 54
  19. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  20. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  21. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  22. 2021Learning Signal-Agnostic Manifolds of Neural Fieldscited by 45
  23. 2021CDPAM: Contrastive learning for perceptual audio similaritycited by 42
  24. 2023Novel-View Acoustic Synthesiscited by 33
  25. 2024Tactile-Augmented Radiance Fieldscited by 26
  26. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  27. 2024Modeling and Driving Human Body Soundfields through Acoustic Primitivescited by 12
  28. 2023Sounding Bodies: Modeling 3D Spatial Sound of Humans Using Body Pose and Audiocited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SiPhy: Single-Image Physical Property Reasoningcited by 2
  2. 2026Objects as Audio-Visual Modal Sound Fieldscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Generating Holistic 3D Human Motion from Speech
  2. 2026Track, Articulate, Act: Generating Articulation from Casual Human Videos
  3. 2025Visual Acoustic Fields
  4. 2025Re-HOLD: Video Hand Object Interaction Reenactment via adaptive Layout-instructed Diffusion Model
  5. 2023Affordance Diffusion: Synthesizing Hand-Object Interactions
  6. 2023ContactArt: Learning 3D Interaction Priors for Category-level Articulated Object and Hand Poses Estimation
  7. 2024From Audio to Photoreal Embodiment: Synthesizing Humans in Conversations
  8. 2024HOIDiffusion: Generating Realistic 3D Hand-Object Interaction Data