mlsanityData to 7 Oct 2026

Sound to Visual Scene Generation by Audio-to-Visual Latent Alignment

arXiv 2303.17490 ↗First posted 30 Mar 2023cs.CV cs.MM cs.SDcited by 41

Not on the map. Why?

Abstract (arXiv)

How does audio describe the world around us? In this paper, we propose a method for generating an image of a scene from sound. Our method addresses the challenges of dealing with the large gaps that often exist between sight and sound. We design a model that works by scheduling the learning procedure of each model component to associate audio-visual modalities despite their information gaps. The key idea is to enrich the audio features with visual information by learning to align audio to visual latent space. We translate the input audio to visual features, then use a pre-trained generator to produce an image. To further improve the quality of our generated images, we use sound source localization to select the audio-visual pairs that have strong cross-modal correlations. We obtain substantially better results on the VEGAS and VGGSound datasets than prior approaches. We also show that we can control our model's predictions by applying simple manipulations to the input waveform, or to the latent space.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 65 of its 77 references that are in the map. First posted on arXiv 30 Mar 2023.

Builds on

65 of 77 references.

References · 65

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017Neural Discrete Representation Learningcited by 4,650
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2016Improved Techniques for Training GANscited by 2,938
  12. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
38 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2017The Kinetics Human Action Video Datasetcited by 1,894
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  5. 2020Knowledge Distillation: A Surveycited by 1,578
  6. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  7. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  8. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  9. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  11. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  12. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  13. 2017Look, Listen and Learncited by 452
  14. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  15. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  16. 2018Image Generation from Scene Graphscited by 324
  17. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  18. 2017Objects that Soundcited by 254
  19. 2016SoundNet: Learning Sound Representations from Unlabeled Videocited by 244
  20. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  21. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  22. 2019Learning Individual Styles of Conversational Gesturecited by 190
  23. 2018Learning to Localize Sound Source in Visual Scenescited by 188
  24. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  25. 2021Localizing Visual Sounds the Hard Waycited by 169
  26. 2021Taming Visually Guided Sound Generationcited by 136
  27. 2015Visually Indicated Soundscited by 128
  28. 2021Instance-Conditioned GANcited by 119
  29. 2019Co-Separating Sounds of Visual Objectscited by 110
  30. 2018Image Generation from Layoutcited by 105
  31. 2020Discriminative Sounding Objects Localization via Self-supervised Audiovisual Matchingcited by 104
  32. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  33. 2021Robust Audio-Visual Instance Discriminationcited by 80
  34. 2016Ambient Sound Provides Supervision for Visual Learningcited by 77
  35. 2017Learning Sight from Sound: Ambient Sound Provides Supervision for Visual Learningcited by 67
  36. 2020Generating Visually Aligned Sound from Videoscited by 67
  37. 2017Deep Cross-Modal Audio-Visual Generationcited by 63
  38. 2019Self-supervised Moving Vehicle Tracking with Stereo Soundcited by 63

50 most cited of 65 All 65 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 17; 2025: 18; 2026: 2.

41 papers.

Sort:

By area:

  1. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  2. 2024Binding Touch to Everything: Learning Unified Multimodal Tactile Representationscited by 94
  3. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  4. 2023Connecting Multi-modal Contrastive Representationscited by 36
  5. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  6. 2024Audio-Synchronized Visual Animationcited by 32
  7. 2023Sound Source Localization is All about Cross-Modal Alignmentcited by 30
  8. 2024MeLFusion: Synthesizing Music from Image and Language Cues using Diffusion Modelscited by 19
  9. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptationcited by 18
  10. 2024Images that Sound: Composing Images and Sounds on a Single Canvascited by 17
  11. 2024SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Modelscited by 15
  12. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
29 more
  1. 2023TextManiA: Enriching Visual Feature by Text-driven Manifold Augmentationcited by 9
  2. 2024Image Anything: Towards Reasoning-coherent and Training-free Multi-modal Image Generationcited by 9
  3. 2024Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignmentcited by 8
  4. 2024EgoSonics: Generating Synchronized Audio for Silent Egocentric Videoscited by 6
  5. 2025KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animationcited by 6
  6. 2025GenHSI: Controllable Generation of Human-Scene Interaction Videoscited by 6
  7. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5
  8. 2024Visual Echoes: A Simple Unified Transformer for Audio-Visual Generationcited by 4
  9. 2025Sounding that Object: Interactive Object-Aware Image to Audio Generationcited by 3
  10. 2024CLIP-VQDiffusion : Langauge Free Training of Text To Image generation using CLIP and vector quantized diffusion modelcited by 2
  11. 2024Automatic Scene Generation: State-of-the-Art Techniques, Models, Datasets, Challenges, and Future Prospectscited by 2
  12. 2025Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectivescited by 2
  13. 2025MACS: Multi-source Audio-to-image Generation with Contextual Significance and Semantic Alignmentcited by 2
  14. 2025Let Your Video Listen to Your Music!cited by 2
  15. 2025DALR: Dual-level Alignment Learning for Multimodal Sentence Representation Learningcited by 2
  16. 2025DreamAudio: Customized Text-to-Audio Generation with Diffusion Modelscited by 2
  17. 2025Schrodinger Audio-Visual Editor: Object-Level Audiovisual Removalcited by 2
  18. 2023UniBriVL: Robust Universal Representation and Generation of Audio Driven Diffusion Modelscited by 1
  19. 2025Towards Generalization of Tactile Image Generation: Reference-Free Evaluation in a Leakage-Free Settingcited by 1
  20. 2025ESG-Net: Event-Aware Semantic Guided Network for Dense Audio-Visual Event Localizationcited by 1
  21. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 1
  22. 2024Survey: Transformer-based Models in Data Modality Conversioncited by 0
  23. 2025Replicating Human Social Perception in Generative AI: Evaluating the Valence-Dominance Modelcited by 0
  24. 2025Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separatorcited by 0
  25. 2025SounDiT: Geo-Contextual Soundscape-to-Landscape Generationcited by 0
  26. 2025Sat2Sound: A Unified Framework for Zero-Shot Soundscape Mappingcited by 0
  27. 2025Learning from Silence and Noise for Visual Sound Source Localizationcited by 0
  28. 2025SeeingSounds: Learning Audio-to-Visual Alignment via Textcited by 0
  29. 2026Towards Expressive and Faithful Audio-to-Image Generation: A Unified Multimodal Dataset and Synthesis Frameworkcited by 0