mlsanityData to 7 Oct 2026

Vision-Infused Deep Audio Inpainting

arXiv 1910.10997 ↗First posted 24 Oct 2019cs.CV cs.LG cs.SDcited by 40

Not on the map. Why?

Abstract (arXiv)

Multi-modality perception is essential to develop interactive intelligence. In this work, we consider a new task of visual information-infused audio inpainting, ie synthesizing missing audio segments that correspond to their accompanying videos. We identify two key aspects for a successful inpainter: (1) It is desirable to operate on spectrograms instead of raw audios. Recent advances in deep semantic image inpainting could be leveraged to go beyond the limitations of traditional audio inpainting. (2) To synthesize visually indicated audio, a visual-audio joint feature space needs to be learned with synchronization of audio and video. To facilitate a large-scale study, we collect a new multi-modality instrument-playing dataset called MUSIC-Extra-Solo (MUSICES) by enriching MUSIC dataset. Extensive experiments demonstrate that our framework is capable of inpainting realistic and varying audio segments with or without visual contexts. More importantly, our synthesized audio segments are coherent with their video counterparts, showing the effectiveness of our proposed Vision-Infused Audio Inpainter (VIAI). Code, models, dataset and video results are available at https://hangz-nju-cuhk.github.io/projects/AudioInpainting

Key facts. Cited by 40 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 12; 2022: 13; 2023: 7; 2024: 5; 2025: 3.

40 papers.

Sort:

By area:

  1. 2021Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representationcited by 288
  2. 2022Learning to Answer Questions in Dynamic Audio-Visual Scenarioscited by 188
  3. 2022SoundSpaces 2.0: A Simulation Platform for Visual-Acoustic Learningcited by 93
  4. 2023AUDIT: Audio Editing by Following Instructions with Latent Diffusion Modelscited by 81
  5. 2021Distilling Audio-Visual Knowledge by Compositional Contrastive Learningcited by 53
  6. 2021Exploiting Explanations for Model Inversion Attackscited by 48
  7. 2022Few-Shot Audio-Visual Learning of Environment Acousticscited by 48
  8. 2021Visually Informed Binaural Audio Generation without Binaural Audioscited by 46
  9. 2022Visual Acoustic Matchingcited by 45
  10. 2023Novel-View Acoustic Synthesiscited by 33
  11. 2021End-to-End Video-To-Speech Synthesis using Generative Adversarial Networkscited by 27
  12. 2022Audio-visual Generalised Zero-shot Learning with Cross-modal Attention and Languagecited by 27
28 more
  1. 2021Can audio-visual integration strengthen robustness under multimodal attacks?cited by 23
  2. 2021Geometry-Aware Multi-Task Learning for Binaural Audio Generation from Videocited by 22
  3. 2023Physics-Driven Diffusion Models for Impact Sound Synthesis from Videoscited by 21
  4. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  5. 2022Visual Sound Localization in the Wild by Cross-Modal Interference Erasingcited by 19
  6. 2022Temporal and cross-modal attention for audio-visual zero-shot learningcited by 19
  7. 2024Arrange, Inpaint, and Refine: Steerable Long-term Music Audio Generation and Editing via Content-based Controlscited by 17
  8. 2021Exploiting Audio-Visual Consistency with Partial Supervision for Spatial Audio Generationcited by 14
  9. 2023Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videoscited by 8
  10. 2025VGGSounder: Audio-Visual Evaluations for Foundation Modelscited by 8
  11. 2022Sound and Visual Representation Learning with Multiple Pretraining Taskscited by 5
  12. 2024Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classificationcited by 4
  13. 2021The Boombox: Visual Reconstruction from Acoustic Vibrationscited by 3
  14. 2022Learning Speaker-specific Lip-to-Speech Generationcited by 3
  15. 2024MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processingcited by 3
  16. 2025SoundVista: Novel-View Ambient Sound Synthesis via Visual-Acoustic Bindingcited by 3
  17. 2025Token-Based Audio Inpainting via Discrete Diffusioncited by 3
  18. 2021HCMS: Hierarchical and Conditional Modality Selection for Efficient Video Recognitioncited by 2
  19. 2021Where and When: Space-Time Attention for Audio-Visual Explanationscited by 1
  20. 2022Audio-Visual Fusion Layers for Event Type Aware Video Recognitioncited by 1
  21. 2022Correcting Mispronunciations in Speech using Spectrogram Inpaintingcited by 1
  22. 2023RobustL2S: Speaker-Specific Lip-to-Speech Synthesis exploiting Self-Supervised Representationscited by 1
  23. 2023Text-to-feature diffusion for audio-visual few-shot learningcited by 1
  24. 2022V-LinkNet: Learning Contextual Inpainting Across Latent Space of Generative Adversarial Networkcited by 0
  25. 2022Show Me Your Face, And I'll Tell You How You Speakcited by 0
  26. 2023Enhancing Gappy Speech Audio Signals with Generative Adversarial Networkscited by 0
  27. 2024Robust Multi-Modal Speech In-Painting: A Sequence-to-Sequence Approachcited by 0
  28. 2024Sequence-to-Sequence Multi-Modal Speech In-Paintingcited by 0