mlsanityData to 7 Oct 2026

Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Aligners

arXiv 2402.17723 ↗First posted 27 Feb 2024cs.CV cs.MM cs.SDcited by 111

Not on the map. Why?

Cited by 28 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Video and audio content creation serves as the core technique for the movie industry and professional users. Recently, existing diffusion-based methods tackle video and audio generation separately, which hinders the technique transfer from academia to industry. In this work, we aim at filling the gap, with a carefully designed optimization-based framework for cross-visual-audio and joint-visual-audio generation. We observe the powerful generation ability of off-the-shelf video or audio generation models. Thus, instead of training the giant models from scratch, we propose to bridge the existing strong models with a shared latent representation space. Specifically, we propose a multimodality latent aligner with the pre-trained ImageBind model. Our latent aligner shares a similar core as the classifier guidance that guides the diffusion denoising process during inference time. Through carefully designed optimization strategy and loss functions, we show the superior performance of our method on joint video-audio generation, visual-steered audio generation, and audio-steered visual generation tasks. The project website can be found at https://yzxing87.github.io/Seeing-and-Hearing/

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). Builds on 46 of its 50 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

46 of 50 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  6. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  7. 2023Qwen Technical Reportcited by 3,273
  8. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  9. 2022Video Diffusion Modelscited by 2,280
  10. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  11. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  12. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
34 more
  1. 2023AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuningcited by 1,418
  2. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  6. 2022MagicVideo: Efficient Video Generation With Latent Diffusion Modelscited by 463
  7. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  8. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  9. 2023Show-1: Marrying Pixel and Latent Diffusion Models for Text-to-Video Generationcited by 300
  10. 2022AudioGen: Textually Guided Audio Generationcited by 292
  11. 2023Preserve Your Own Correlation: A Noise Prior for Video Diffusion Modelscited by 284
  12. 2023Emu: Enhancing Image Generation Models Using Photogenic Needles in a Haystackcited by 278
  13. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generationcited by 256
  14. 2022Long Video Generation with Time-Agnostic VQGAN and Time-Sensitive Transformercited by 244
  15. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 225
  16. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  17. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 210
  18. 2023Any-to-Any Generation via Composable Diffusioncited by 203
  19. 2023Text-to-Audio Generation using Instruction-Tuned LLM and Latent Diffusion Modelcited by 145
  20. 2023Latent-Shift: Latent Diffusion with Temporal Shift for Efficient Text-to-Video Generationcited by 137
  21. 2021Taming Visually Guided Sound Generationcited by 136
  22. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 133
  23. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  24. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generationcited by 90
  25. 2023Animate-A-Story: Storytelling with Retrieval-Augmented Video Generationcited by 89
  26. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
  27. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  28. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  29. 2023MovieFactory: Automatic Movie Creation from Text using Large Generative Models for Language and Imagescited by 42
  30. 2023Sound to Visual Scene Generation by Audio-to-Visual Latent Alignmentcited by 41
  31. 2020Sound2Sight: Generating Visual Dynamics from Sound and Contextcited by 28
  32. 2023Physics-Driven Diffusion Models for Impact Sound Synthesis from Videoscited by 21
  33. 2023CLIPSonic: Text-to-Audio Synthesis with Unlabeled Videos and Pretrained Language-Vision Modelscited by 20
  34. 2021Audio-to-Image Cross-Modal Generationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 53; 2026: 28.

111 papers.

Sort:

By area:

  1. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  2. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  3. 2025JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronizationcited by 84
  4. 2024Sora as a World Model? A Complete Survey on Text-to-Video Generationcited by 53
  5. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 52
  6. 2024Inference-Time Alignment of Diffusion Models with Direct Noise Optimizationcited by 51
  7. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  8. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  9. 2024VidMuse: A Simple Video-to-Music Generation Framework with Long-Short-Term Modelingcited by 39
  10. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generationcited by 39
  11. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generationcited by 38
  12. 2024LLMs Meet Multimodal Generation and Editing: A Surveycited by 35
38 more
  1. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  2. 2024Foundation Models for Music: A Surveycited by 34
  3. 2024TAVGBench: Benchmarking Text to Audible-Video Generationcited by 33
  4. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  5. 2024Taming Data and Transformers for Audio Generationcited by 28
  6. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  7. 2024Both Ears Wide Open: Towards Language-Driven Spatial Audio Generationcited by 25
  8. 2025AudioX: A Unified Framework for Anything-to-Audio Generationcited by 25
  9. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
  10. 2026JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generationcited by 22
  11. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  12. 2025UniForm: A Unified Multi-Task Diffusion Transformer for Audio-Video Generationcited by 21
  13. 2024A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generationcited by 17
  14. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generationcited by 17
  15. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  16. 2024VinTAGe: Joint Video and Text Conditioning for Holistic Audio Generationcited by 16
  17. 2025Audio-Sync Video Generation with Multi-Stream Temporal Controlcited by 14
  18. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generationcited by 12
  19. 2025AudioGen-Omni: A Unified Multimodal Diffusion Transformer for Video-Synchronized Audio, Speech, and Song Generationcited by 11
  20. 2024Deepfake Media Generation and Detection in the Generative AI Era: A Survey and Outlookcited by 10
  21. 2025Synchronized Video-to-Audio Generation via Mel Quantization-Continuum Decompositioncited by 10
  22. 2025TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesiscited by 10
  23. 2026Audio-Omni: Extending Multi-modal Understanding to Versatile Audio Generation and Editingcited by 10
  24. 2024TALE: Training-free Cross-domain Image Composition via Adaptive Latent Manipulation and Energy-guided Optimizationcited by 9
  25. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  26. 2025Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interactioncited by 9
  27. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  28. 2025Long-Video Audio Synthesis with Multi-Agent Collaborationcited by 8
  29. 2025Pulp Motion: Framing-aware multimodal camera and human motion generationcited by 8
  30. 2024Draw an Audio: Leveraging Multi-Instruction for Video-to-Audio Synthesiscited by 7
  31. 2025Language Model Based Text-to-Audio Generation: Anti-Causally Aligned Collaborative Residual Transformerscited by 7
  32. 2026MM-Sonate: Multimodal Controllable Audio-Video Generation with Zero-Shot Voice Cloningcited by 7
  33. 2025RFLAV: Rolling Flow matching for infinite Audio Video generationcited by 6
  34. 2025Projected Coupled Diffusion for Test-Time Constrained Joint Generationcited by 6
  35. 2025FoleyBench: A Benchmark For Video-to-Audio Modelscited by 6
  36. 2026OmniNFT: Modality-wise Omni Diffusion Reinforcement for Joint Audio-Video Generationcited by 6
  37. 2024AudioScenic: Audio-Driven Video Scene Editingcited by 5
  38. 2024Sound2Vision: Generating Diverse Visuals from Audio through Cross-Modal Latent Alignmentcited by 5

50 most cited of 111 All 111 (JSON)