mlsanityData to 14 Oct 2026

AdVerb: Visually Guided Audio Dereverberation

arXiv 2308.12370 ↗First posted 23 Aug 2023cs.CV cs.MM cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

We present AdVerb, a novel audio-visual dereverberation framework that uses visual cues in addition to the reverberant sound to estimate clean audio. Although audio-only dereverberation is a well-studied problem, our approach incorporates the complementary visual modality to perform audio dereverberation. Given an image of the environment where the reverberated sound signal has been recorded, AdVerb employs a novel geometry-aware cross-modal transformer architecture that captures scene geometry and audio-visual cross-modal relationship to generate a complex ideal ratio mask, which, when applied to the reverberant audio predicts the clean sound. The effectiveness of our method is demonstrated through extensive quantitative and qualitative evaluations. Our approach significantly outperforms traditional audio-only and audio-visual baselines on three downstream tasks: speech enhancement, speech recognition, and speaker verification, with relative improvements in the range of 18% - 82% on the LibriSpeech test-clean set. We also achieve highly satisfactory RT60 error scores on the AVSpeech dataset.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 60 of its 95 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

60 of 95 references.

References · 60

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  7. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  8. 2017Matterport3D: Learning from RGB-D Data in Indoor Environmentscited by 1,376
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,154
  10. 2020A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wildcited by 672
  11. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 425
  12. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 406
38 more
  1. 2021AD-NeRF: Audio Driven Neural Radiance Fields for Talking Head Synthesiscited by 339
  2. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  3. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  4. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  5. 2018The Sound of Pixelscited by 259
  6. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 188
  7. 2020Real Time Speech Enhancement in the Waveform Domaincited by 186
  8. 2021Audio2Head: Audio-driven One-shot Talking-head Generation with Natural Head Motioncited by 149
  9. 2019The Sound of Motionscited by 146
  10. 2018Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps with Accurate Object Boundariescited by 145
  11. 2015Visually Indicated Soundscited by 128
  12. 2021MetricGAN+: An Improved Version of MetricGAN for Speech Enhancementcited by 114
  13. 2021Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detectioncited by 113
  14. 2020Music Gesture for Visual Sound Separationcited by 112
  15. 2020Talking-head Generation with Rhythmic Head Motioncited by 112
  16. 2019Co-Separating Sounds of Visual Objectscited by 110
  17. 2018The Conversation: Deep Audio-Visual Speech Enhancementcited by 109
  18. 2019HorizonNet: Learning Room Layout with 1D Representation and Pano Stretch Data Augmentationcited by 108
  19. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 95
  20. 2019AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detectioncited by 83
  21. 2020Semantic Audio-Visual Navigationcited by 82
  22. 2020An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separationcited by 81
  23. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  24. 2020VisualEchoes: Spatial Image Representation Learning through Echolocationcited by 55
  25. 2019Recursive Visual Sound Separation Using Minus-Plus Netcited by 53
  26. 2022Sound Adversarial Audio-Visual Navigationcited by 48
  27. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  28. 2022Visual Acoustic Matchingcited by 45
  29. 2018Building and Evaluation of a Real Room Impulse Response Datasetcited by 43
  30. 2021LED2-Net: Monocular 360 Layout Estimation via Differentiable Depth Renderingcited by 43
  31. 2020Active Speakers in Contextcited by 41
  32. 2020HiFi-GAN: High-Fidelity Denoising and Dereverberation Based on Speech Deep Features in Adversarial Networkscited by 40
  33. 2022VoiceFixer: A Unified Framework for High-Fidelity Speech Restorationcited by 37
  34. 2021FAST-RIR: Fast neural diffuse room impulse response generatorcited by 35
  35. 2021Image2Reverb: Cross-Modal Reverb Impulse Response Synthesiscited by 34
  36. 2019My lips are concealed: Audio-visual speech enhancement through obstructionscited by 26
  37. 2021Learning Audio-Visual Dereverberationcited by 26
  38. 2019Pano Popups: Indoor 3D Reconstruction with a Plane-Aware Networkcited by 25

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 6; 2025: 6; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023AV-RIR: Audio-Visual Room Impulse Response Estimationcited by 20
  2. 2024Real Acoustic Fields: An Audio-Visual Room Acoustics Dataset and Benchmarkcited by 20
  3. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  4. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  5. 2025MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystackscited by 12
  6. 2024SEE-2-SOUND: Zero-Shot Spatial Environment-to-Spatial Soundcited by 10
  7. 2024NeRAF: 3D Scene Infused Neural Radiance and Acoustic Fieldscited by 9
  8. 2025Hearing Anywhere in Any Environmentcited by 8
  9. 2025Differentiable Room Acoustic Rendering with Multi-View Vision Priorscited by 5
  10. 2025EgoAdapt: Adaptive Multisensory Distillation and Policy Learning for Efficient Egocentric Perceptioncited by 4
  11. 2025How Would It Sound? Material-Controlled Multimodal Acoustic Profile Generation for Indoor Scenescited by 4
  12. 2025AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understandingcited by 4
3 more
  1. 2024SOAF: Scene Occlusion-aware Neural Acoustic Fieldcited by 3
  2. 2026Few-shot Acoustic Synthesis with Multimodal Flow Matchingcited by 2
  3. 2026Materialistic RIR: Material Conditioned Realistic RIR Generationcited by 0