mlsanityData to 7 Oct 2026

Images that Sound: Composing Images and Sounds on a Single Canvas

arXiv 2405.12221 ↗First posted 20 May 2024cs.CV cs.LG cs.MMcited by 17

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Spectrograms are 2D representations of sound that look very different from the images found in our visual world. And natural images, when played as spectrograms, make unnatural sounds. In this paper, we show that it is possible to synthesize spectrograms that simultaneously look like natural images and sound like natural audio. We call these visual spectrograms images that sound. Our approach is simple and zero-shot, and it leverages pre-trained text-to-image and text-to-spectrogram diffusion models that operate in a shared latent space. During the reverse process, we denoise noisy latents with both the audio and image diffusion models in parallel, resulting in a sample that is likely under both models. Through quantitative evaluations and perceptual studies, we find that our method successfully generates spectrograms that align with a desired audio prompt while also taking the visual appearance of a desired image prompt. Please see our project page for video results: https://ificl.github.io/images-that-sound/

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 99 of its 122 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

99 of 122 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization15 Apr
  2. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation2 Jan
  3. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  4. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Model24 Apr
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan

Other references · 94

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  8. 2022Classifier-Free Diffusion Guidancecited by 5,717
  9. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,067
  10. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  11. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  2. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  3. 2022Video Diffusion Modelscited by 2,280
  4. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,170
  5. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,903
  6. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  7. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  8. 2023Zero-1-to-3: Zero-shot One Image to 3D Objectcited by 1,483
  9. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
  10. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  11. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  12. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  13. 2022Denoising Diffusion Restoration Modelscited by 951
  14. 2021Diffusion Probabilistic Models for 3D Point Cloud Generationcited by 734
  15. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  16. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  17. 2021ILVR: Conditioning Method for Denoising Diffusion Probabilistic Modelscited by 634
  18. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  19. 2022Zero-Shot Image Restoration Using Denoising Diffusion Null-Space Modelcited by 574
  20. 2023MultiDiffusion: Fusing Diffusion Paths for Controlled Image Generationcited by 529
  21. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  22. 2017Look, Listen and Learncited by 452
  23. 2024Lumiere: A Space-Time Diffusion Model for Video Generationcited by 450
  24. 2024CAT3D: Create Anything in 3D with Multi-View Diffusion Modelscited by 425
  25. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  26. 2023Diffusion Self-Guidance for Controllable Image Generationcited by 305
  27. 2017Objects that Soundcited by 254
  28. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  29. 2023Text2Room: Extracting Textured 3D Meshes from 2D Text-to-Image Modelscited by 245
  30. 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
  31. 2023Reduce, Reuse, Recycle: Compositional Generation with Energy-Based Diffusion Models and MCMCcited by 216
  32. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  33. 2020Self-Supervised Learning of Audio-Visual Objects from Videocited by 186
  34. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  35. 2019Listen to Look: Action Recognition by Previewing Audiocited by 163
  36. 2019The Sound of Motionscited by 145
  37. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  38. 2021Taming Visually Guided Sound Generationcited by 136

50 most cited of 94 All 94 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 8; 2026: 5.

17 papers.

Sort:

By area:

  1. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  2. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  3. 2024VMAS: Video-to-Music Generation via Semantic Alignment in Web Music Videoscited by 14
  4. 2025Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoisingcited by 13
  5. 2025StochSync: Stochastic Diffusion Synchronization for Image Generation in Arbitrary Spacescited by 8
  6. 2025Unconditional Priors Matter! Improving Conditional Generation of Fine-Tuned Diffusion Modelscited by 7
  7. 2025LookingGlass: Generative Anamorphoses via Laplacian Pyramid Warpingcited by 7
  8. 2025GPS as a Control Signal for Image Generationcited by 5
  9. 2026V2M-Zero: Zero-Pair Time-Aligned Video-to-Music Generationcited by 2
  10. 2026Stroke of Surprise: Progressive Semantic Illusions in Vector Sketchingcited by 1
  11. 2024Making Images from Images: Interleaving Denoising and Transformationcited by 0
  12. 2025StableMotion: One-Step Motion Estimation with Diffusion Priorcited by 0
5 more
  1. 2025SpeechOp: Inference-Time Task Composition for Generative Speech Processingcited by 0
  2. 2025StereoSync: Spatially-Aware Stereo Audio Generation from Videocited by 0
  3. 2026JanusMesh: Fast and Zero-Shot 3D Visual Illusion Generation via Cross-Space Denoisingcited by 0
  4. 2026Asynchronous Multimodal Diffusion Policy Composition via Latency-Aware Guidance Fusioncited by 0
  5. 2026Supervising Sound Localization by In-the-wild Egomotioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Soundini: Sound-Guided Diffusion for Natural Video Editing
  2. 2024SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models
  3. 2025Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation
  4. 2026Dual-View Predictive Diffusion: Lightweight Speech Enhancement via Spectrogram-Image Synergy
  5. 2024Repurposing Image Diffusion Models for Training-Free Music Style Transfer on Mel-spectrograms
  6. 2023Align, Adapt and Inject: Sound-guided Unified Image Generation
  7. 2024SoundBrush: Sound as a Brush for Visual Scene Editing
  8. 2024SoundMorpher: Perceptually-Uniform Sound Morphing with Diffusion Model