mlsanityData to 7 Oct 2026

Visual Acoustic Matching

arXiv 2202.06875 ↗First posted 14 Feb 2022cs.CV cs.MM cs.SDcited by 45

Not on the map. Why?

Abstract (arXiv)

We introduce the visual acoustic matching task, in which an audio clip is transformed to sound like it was recorded in a target environment. Given an image of the target environment and a waveform for the source audio, the goal is to re-synthesize the audio to match the target room acoustics as suggested by its visible geometry and materials. To address this novel task, we propose a cross-modal transformer model that uses audio-visual attention to inject visual properties into the audio and generate realistic audio output. In addition, we devise a self-supervised training objective that can learn acoustic matching from in-the-wild Web videos, despite their lack of acoustically mismatched audio. We demonstrate that our approach successfully translates human speech to a variety of real-world environments depicted in images, outperforming both traditional acoustic matching and more heavily supervised baselines.

Key facts. Cited by 45 papers in the map (data through 7 Oct 2026). Builds on 44 of its 82 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

44 of 82 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  5. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  7. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  8. 2016Least Squares Generative Adversarial Networkscited by 1,000
  9. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  10. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  11. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 477
  12. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
32 more
  1. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 351
  2. 2019Parallel WaveGAN: A fast waveform generation model based on generative adversarial networks with multi-resolution spectrogramcited by 337
  3. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  4. 2018WaveGlow: A Flow-based Generative Network for Speech Synthesiscited by 308
  5. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  6. 2018The Sound of Pixelscited by 258
  7. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  8. 2021Anticipative Video Transformercited by 197
  9. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  10. 2019The Sound of Motionscited by 145
  11. 2019MOSNet: Deep Learning based Objective Assessment for Voice Conversioncited by 126
  12. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  13. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 107
  14. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  15. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  16. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  17. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  18. 2020Semantic Audio-Visual Navigationcited by 79
  19. 2019Phase-aware Speech Enhancement with Deep Complex U-Netcited by 73
  20. 20182.5D Visual Soundcited by 63
  21. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  22. 2019Vision-Infused Deep Audio Inpaintingcited by 40
  23. 2017Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networkscited by 36
  24. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  25. 2021The Right to Talk: An Audio-Visual Transformer Approachcited by 33
  26. 2021Space-Time Crop & Attend: Improving Cross-modal Video Representation Learningcited by 31
  27. 2019Audio-visual Speech Enhancement Using Conditional Variational Auto-Encoderscited by 27
  28. 2021Learning Audio-Visual Dereverberationcited by 26
  29. 2021Filtered Noise Shaping for Time Domain Room Impulse Response Estimation From Reverberant Speechcited by 16
  30. 2019A Neural Vocoder with Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesiscited by 10
  31. 2021Reverb Conversion of Mixed Vocal Tracks Using an End-to-end Convolutional Deep Neural Networkcited by 8
  32. 2020Dereverberation using joint estimation of dry speech signal and acoustic systemcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 14; 2024: 17; 2025: 6; 2026: 3.

45 papers.

Sort:

By area:

  1. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  2. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  3. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  4. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  5. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 40
  6. 2024MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D Worldcited by 38
  7. 2023Novel-View Acoustic Synthesiscited by 33
  8. 2024EgoChoir: Capturing 3D Human-Object Interaction Regions from Egocentric Viewscited by 26
  9. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformercited by 24
  10. 2022MESH2IR: Neural Acoustic Impulse Response Generator for Complex 3D Scenescited by 23
  11. 2023Measuring Acoustics with Collaborative Multiple Agentscited by 21
  12. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
33 more
  1. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  2. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  3. 2022Pay Self-Attention to Audio-Visual Navigationcited by 17
  4. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  5. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  6. 2023Self-Supervised Visual Acoustic Matchingcited by 14
  7. 2024Hearing Anything Anywherecited by 12
  8. 2024SoundingActions: Learning How Actions Sound from Narrated Egocentric Videoscited by 11
  9. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  10. 2023Chat2Map: Efficient Scene Mapping from Multi-Ego Conversationscited by 8
  11. 2023RealImpact: A Dataset of Impact Sound Fields for Real Objectscited by 8
  12. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  13. 2024NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fieldscited by 8
  14. 2024Self-Supervised Audio-Visual Soundscape Stylizationcited by 8
  15. 2025Hearing Anywhere in Any Environmentcited by 8
  16. 2024SAV-SE: Scene-aware Audio-Visual Speech Enhancement with Selective State Space Modelcited by 6
  17. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  18. 2024Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speechcited by 4
  19. 2025How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenescited by 4
  20. 2023Looking Similar, Sounding Different: Leveraging Counterfactual Cross-Modal Pairs for Audiovisual Representation Learningcited by 3
  21. 2024Gaussian Flow Bridges for Audio Domain Transfer with Unpaired Datacited by 3
  22. 2024SOAF: Scene Occlusion-aware Neural Acoustic Fieldcited by 3
  23. 2025SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Bindingcited by 3
  24. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  25. 2024ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modelingcited by 2
  26. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  27. 2022One-Shot Acoustic Matching Of Audio Signals -- Learning to Hear Music In Any Room/ Concert Hallcited by 0
  28. 2024Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusioncited by 0
  29. 2024ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgationcited by 0
  30. 2024AVS-Net: Audio-Visual Scale Net for Self-supervised Monocular Metric Depth Estimationcited by 0
  31. 2025AudioScene: Integrating Object-Event Audio into 3D Scenescited by 0
  32. 2026Materialistic RIR: Material Conditioned Realistic RIR Generationcited by 0
  33. 2026MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimationcited by 0