# Objects as Audio-Visual Modal Sound Fields

arXiv: https://arxiv.org/abs/2608.05145 · HTML: https://mlsanity.com/paper/2608.05145/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 49 references that are in the map.
- First posted on arXiv 5 Aug 2026.

## Builds on (most cited first, 30 of 31)

- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [3D Gaussian Splatting for Real-Time Radiance Field Rendering](https://mlsanity.com/paper/2308.04079/) (2023)
- [Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields](https://mlsanity.com/paper/2103.13415/) (2021)
- [NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction](https://mlsanity.com/paper/2106.10689/) (2021)
- [2D Gaussian Splatting for Geometrically Accurate Radiance Fields](https://mlsanity.com/paper/2403.17888/) (2024)
- [SuGaR: Surface-Aligned Gaussian Splatting for Efficient 3D Mesh Reconstruction and High-Quality Mesh Rendering](https://mlsanity.com/paper/2311.12775/) (2023)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [Vox-E: Text-guided Voxel Editing of 3D Objects](https://mlsanity.com/paper/2303.12048/) (2023)
- [ObjectFolder 2.0: A Multisensory Object Dataset for Sim2Real Transfer](https://mlsanity.com/paper/2204.02389/) (2022)
- [ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations](https://mlsanity.com/paper/2109.07991/) (2021)
- [See, Hear, and Feel: Smart Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2212.03858/) (2022)
- [3D Shape Reconstruction from Vision and Touch](https://mlsanity.com/paper/2007.03778/) (2020)
- [CDPAM: Contrastive learning for perceptual audio similarity](https://mlsanity.com/paper/2102.05109/) (2021)
- [The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objects](https://mlsanity.com/paper/2306.00956/) (2023)
- [Tailor3D: Customized 3D Assets Editing and Generation with Dual-Side Images](https://mlsanity.com/paper/2407.06191/) (2024)
- [ShapeMap 3-D: Efficient shape mapping through dense touch and vision](https://mlsanity.com/paper/2109.09884/) (2021)
- [NANO3D: A Training-Free Approach for Efficient 3D Editing Without Masks](https://mlsanity.com/paper/2510.15019/) (2025)
- [Tactile-Augmented Radiance Fields](https://mlsanity.com/paper/2405.04534/) (2024)
- [Physics-Driven Diffusion Models for Impact Sound Synthesis from Videos](https://mlsanity.com/paper/2303.16897/) (2023)
- [Learning Object-Centric Neural Scattering Functions for Free-Viewpoint Relighting and Scene Composition](https://mlsanity.com/paper/2303.06138/) (2023)
- [RNG: Relightable Neural Gaussians](https://mlsanity.com/paper/2409.19702/) (2024)
- [Localizing the Object Contact through Matching Tactile Features with Visual Map](https://mlsanity.com/paper/1708.04441/) (2017)
- [Tactile Beyond Pixels: Multisensory Touch Representations for Robot Manipulation](https://mlsanity.com/paper/2506.14754/) (2025)
- [RealImpact: A Dataset of Impact Sound Fields for Real Objects](https://mlsanity.com/paper/2306.09944/) (2023)
- [DiffSound: Differentiable Modal Sound Rendering and Inverse Rendering for Diverse Inference Tasks](https://mlsanity.com/paper/2409.13486/) (2024)
- [Score Distillation Sampling for Audio: Source Separation, Synthesis, and Beyond](https://mlsanity.com/paper/2505.04621/) (2025)
- [X-Capture: An Open-Source Portable Device for Multi-Sensory Learning](https://mlsanity.com/paper/2504.02318/) (2025)
- [Sounding that Object: Interactive Object-Aware Image to Audio Generation](https://mlsanity.com/paper/2506.04214/) (2025)
- [Hearing Hands: Generating Sounds from Physical Interactions in 3D Scenes](https://mlsanity.com/paper/2506.09989/) (2025)
- [SonicGauss: Position-Aware Physical Sound Synthesis for 3D Gaussian Representations](https://mlsanity.com/paper/2507.19835/) (2025)

## Abstract (arXiv)

While modern 3D reconstruction excels at modeling object geometry and appearance, it largely ignores the rich acoustic cues revealed through physical interaction. Object impact sounds convey material, stiffness, and structural properties that complement vision, yet existing impact sound modeling approaches either rely on expensive physics-based simulation or require large datasets to generalize in a purely data-driven manner. We introduce Audio-Visual Modal Sound Field (AV-MSF), a novel object-level acoustic representation reconstructed from multi-view images and only a few impact sound recordings. AV-MSF builds on 3D Gaussian Splatting integrated with dense 3D visual feature to provide a strong geometry-aware prior, and represents the impact sound field using compact, physically meaningful modal parameters, enabling robust few-shot reconstruction. Experiments on two real-world datasets show that AV-MSF achieves state-of-the-art impact sound rendering, outperforming both physics-based and data-driven baselines. Furthermore, we demonstrate downstream applications enabled by our representation, including contact localization and object sound editing.