mlsanityData to 7 Oct 2026

Learning Audio-Visual Dereverberation

arXiv 2106.07732 ↗First posted 14 Jun 2021cs.SD cs.CV cs.LGcited by 26

Not on the map. Why?

Abstract (arXiv)

Reverberation not only degrades the quality of speech for human perception, but also severely impacts the accuracy of automatic speech recognition. Prior work attempts to remove reverberation based on the audio modality only. Our idea is to learn to dereverberate speech from audio-visual observations. The visual environment surrounding a human speaker reveals important cues about the room geometry, materials, and speaker location, all of which influence the precise reverberation effects. We introduce Visually-Informed Dereverberation of Audio (VIDA), an end-to-end approach that learns to remove reverberation based on both the observed monaural sound and visual scene. In support of this new task, we develop a large-scale dataset SoundSpaces-Speech that uses realistic acoustic renderings of speech in real-world 3D scans of homes offering a variety of room acoustics. Demonstrating our approach on both simulated and real imagery for speech enhancement, speech recognition, and speaker identification, we show it achieves state-of-the-art performance and substantially improves over audio-only methods.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 39 of its 78 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

39 of 78 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,354
  7. 2014Depth Map Prediction from a Single Image using a Multi-Scale Deep Networkcited by 1,264
  8. 2019The Replica Dataset: A Digital Replica of Indoor Spacescited by 885
  9. 2017VoxCeleb: a large-scale speaker identification datasetcited by 714
  10. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  11. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  12. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
27 more
  1. 2018Cooperative Learning of Audio and Video Models from Self-Supervised Synchronizationcited by 246
  2. 2015Deep clustering: Discriminative embeddings for segmentation and separationcited by 211
  3. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  4. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  5. 2019The Sound of Motionscited by 145
  6. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  7. 2021VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistencycited by 114
  8. 2021MetricGAN+: An Improved Version of MetricGAN for Speech Enhancementcited by 114
  9. 2019MetricGAN: Generative Adversarial Networks based Black-box Metric Scores Optimization for Speech Enhancementcited by 104
  10. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  11. 2019Look, Listen, and Act: Towards Audio-Visual Embodied Navigationcited by 95
  12. 2020Learning Representations from Audio-Visual Spatial Alignmentcited by 95
  13. 2018Self-Supervised Generation of Spatial Audio for 360 Videocited by 93
  14. 2020Semantic Audio-Visual Navigationcited by 79
  15. 20182.5D Visual Soundcited by 63
  16. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 54
  17. 2018Building and Evaluation of a Real Room Impulse Response Datasetcited by 43
  18. 2020See, Hear, Explore: Curiosity via Audio-Visual Associationcited by 43
  19. 2020HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networkscited by 40
  20. 2021Move2Hear: Active Audio-Visual Source Separationcited by 35
  21. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  22. 2018Voices Obscured in Complex Environmental Settings (VOICES) corpuscited by 28
  23. 2018Scene-Aware Audio for 360textdegree Videoscited by 25
  24. 2018Speech Dereverberation Using Fully Convolutional Networkscited by 14
  25. 2019Audio-Visual Speech Separation and Dereverberation with a Two-Stage Multimodal Networkcited by 13
  26. 2017Adversarial Semi-Supervised Audio Source Separation applied to Singing Voice Extractioncited by 11
  27. 2020Listening to Sounds of Silence for Speech Denoisingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 8; 2024: 8; 2025: 2; 2026: 2.

26 papers.

Sort:

By area:

  1. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  2. 2023AV-NeRF: Learning Neural Fields for Real-World Audio-Visual Scene Synthesiscited by 49
  3. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  4. 2022Visual Acoustic Matchingcited by 45
  5. 2021Catch Me If You Hear Me: Audio-Visual Navigation in Complex Unmapped Environments with Moving Soundscited by 40
  6. 2023Novel-View Acoustic Synthesiscited by 33
  7. 2023ViT-TTS: Visual Text-to-Speech with Scalable Diffusion Transformercited by 24
  8. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
  9. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  10. 2022Pay Self-Attention to Audio-Visual Navigationcited by 17
  11. 2023AdVerb: Visually Guided Audio Dereverberationcited by 15
  12. 2023Self-Supervised Visual Acoustic Matchingcited by 14
14 more
  1. 2024Hearing Anything Anywherecited by 12
  2. 2024NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fieldscited by 8
  3. 2023Replay: Multi-modal Multi-view Acted Videos for Casual Holographycited by 6
  4. 2023Novel-View Acoustic Synthesis from 3D Reconstructed Roomscited by 6
  5. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  6. 2024Multi-Source Spatial Knowledge Understanding for Immersive Visual Text-to-Speechcited by 4
  7. 2024SOAF: Scene Occlusion-aware Neural Acoustic Fieldcited by 3
  8. 2025SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Bindingcited by 3
  9. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  10. 2022Dynamical Audio-Visual Navigation: Catching Unheard Moving Sound Sources in Unmapped 3D Environmentscited by 0
  11. 2024Mutual Learning for Acoustic Matching and Dereverberation via Visual Scene-driven Diffusioncited by 0
  12. 2024ANAVI: Audio Noise Awareness using Visuals of Indoor environments for NAVIgationcited by 0
  13. 2024SoundLoc3D: Invisible 3D Sound Source Localization and Classification Using a Multimodal RGB-D Acoustic Cameracited by 0
  14. 2026MMAudioReverbs: Video-Guided Acoustic Modeling for Dereverberation and Room Impulse Response Estimationcited by 0