mlsanityData to 14 Oct 2026

Taming Visually Guided Sound Generation

arXiv 2110.08791 ↗First posted 17 Oct 2021cs.CV cs.AI cs.LGcited by 136

Not on the map. Why?

Cited by 36 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent advances in visually-induced audio generation are based on sampling short, low-fidelity, and one-class sounds. Moreover, sampling 1 second of audio from the state-of-the-art model takes minutes on a high-end GPU. In this work, we propose a single model capable of generating visually relevant, high-fidelity sounds prompted with a set of frames from open-domain videos in less time than it takes to play it on a single GPU. We train a transformer to sample a new spectrogram from the pre-trained spectrogram codebook given the set of video features. The codebook is obtained using a variant of VQGAN trained to produce a compact sampling space with a novel spectrogram-based perceptual loss. The generated spectrogram is transformed into a waveform using a window-based GAN that significantly speeds up generation. Considering the lack of metrics for automatic evaluation of generated spectrograms, we also build a family of metrics called FID and MKL. These metrics are based on a novel sound classifier, called Melception, and designed to evaluate the fidelity and relevance of open-domain samples. Both qualitative and quantitative studies are conducted on small- and large-scale datasets to evaluate the fidelity and relevance of generated samples. We also compare our model to the state-of-the-art and observe a substantial improvement in quality, size, and computation time. Code, demo, and samples: v-iashin.github.io/SpecVQGAN

Key facts. Cited by 136 papers in the map (data through 14 Oct 2026). Builds on 68 of its 84 references that are in the map. First posted on arXiv 17 Oct 2021.

Builds on

68 of 84 references.

References · 68

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  6. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,177
  7. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  8. 2017Neural Discrete Representation Learningcited by 4,712
  9. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  10. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  11. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  12. 2016Improved Techniques for Training GANscited by 2,970
38 more
  1. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  2. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  3. 2018Group Normalizationcited by 1,524
  4. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,387
  5. 2018Demystifying MMD GANscited by 1,293
  6. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  7. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  8. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  9. 2018Self-Attention Generative Adversarial Networkscited by 809
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 771
  11. 2016Generative Adversarial Text to Image Synthesiscited by 735
  12. 2016CNN Architectures for Large-Scale Audio Classificationcited by 704
  13. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 694
  14. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  15. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  17. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 628
  18. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 577
  19. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
  20. 2020Jukebox: A Generative Model for Musiccited by 542
  21. 2017MoCoGAN: Decomposing Motion and Content for Video Generationcited by 524
  22. 2018Video-to-Video Synthesiscited by 390
  23. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  24. 2018Everybody Dance Nowcited by 315
  25. 2018The Sound of Pixelscited by 259
  26. 2018cGANs with Projection Discriminatorcited by 254
  27. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 225
  28. 2018Stochastic Adversarial Video Predictioncited by 173
  29. 2019The Sound of Motionscited by 146
  30. 2015Visually Indicated Soundscited by 128
  31. 2020Music Gesture for Visual Sound Separationcited by 112
  32. 2016SampleRNN: An Unconditional End-to-End Neural Audio Generation Modelcited by 110
  33. 2020Latent Video Transformercited by 106
  34. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  35. 2019High Fidelity Speech Synthesis with Adversarial Networkscited by 92
  36. 2016Learning Dense Correspondence via 3D-guided Cycle Consistencycited by 91
  37. 2020Foley Music: Learning to Generate Music from Videoscited by 83
  38. 2018Benchmark Analysis of Representative Deep Neural Network Architecturescited by 77

50 most cited of 68 All 68 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 18; 2024: 51; 2025: 42; 2026: 18.

136 papers.

Sort:

By area:

  1. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  2. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 324
  3. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 257
  4. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  5. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  6. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  7. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  8. 2023A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AIcited by 68
  9. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  10. 2024Zero-Shot Unsupervised and Text-Based Audio Editing Using DDPM Inversioncited by 60
  11. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  12. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
38 more
  1. 2024Temporally Aligned Audio for Video with Autoregressioncited by 52
  2. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  3. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  4. 2024SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General Soundcited by 45
  5. 2022Learning in Audio-visual Context: A Review, Analysis, and New Perspectivecited by 44
  6. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  7. 2023The ObjectFolder Benchmark: Multisensory Learning with Neural and Real Objectscited by 41
  8. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 40
  9. 2022Quantized GAN for Complex Music Generation from Dance Videoscited by 37
  10. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 34
  11. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  12. 2023Controllable Music Production with Diffusion Models and Guidance Gradientscited by 29
  13. 2024Text-to-Audio Generation Synchronized with Videoscited by 28
  14. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  15. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  16. 2023Retrieval-Augmented Text-to-Audio Generationcited by 24
  17. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  18. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
  19. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  20. 2022Learning Visual Styles from Audio-Visual Associationscited by 22
  21. 2023AudioToken: Adaptation of Text-Conditioned Diffusion Models for Audio-to-Image Generationcited by 22
  22. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 22
  23. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 22
  24. 2024Action2Sound: Ambient-Aware Generation of Action Sounds from Egocentric Videoscited by 21
  25. 2022MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENerationcited by 20
  26. 2023CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Modelscited by 20
  27. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generationcited by 19
  28. 2023Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotationcited by 18
  29. 2024A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generationcited by 18
  30. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  31. 2025Generative AI for Cel-Animation: A Surveycited by 17
  32. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  33. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  34. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  35. 2023Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generationcited by 15
  36. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 15
  37. 2024MuVi: Video-to-Music Generation with Semantic Alignment and Rhythmic Synchronizationcited by 14
  38. 2025Object-AVEdit: An Object-level Audio-Visual Editing Modelcited by 14

50 most cited of 136 All 136 (JSON)