mlsanityData to 7 Oct 2026

AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion

arXiv 2305.04001 ↗First posted 6 May 2023cs.CV cs.AIcited by 19

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Recent advances in diffusion models have showcased promising results in the text-to-video (T2V) synthesis task. However, as these T2V models solely employ text as the guidance, they tend to struggle in modeling detailed temporal dynamics. In this paper, we introduce a novel T2V framework that additionally employ audio signals to control the temporal dynamics, empowering an off-the-shelf T2I diffusion to generate audio-aligned videos. We propose audio-based regional editing and signal smoothing to strike a good balance between the two contradicting desiderata of video synthesis, i.e., temporal flexibility and coherence. We empirically demonstrate the effectiveness of our method through experiments, and further present practical applications for contents creation.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 23 of its 24 references that are in the map. First posted on arXiv 6 May 2023.

Builds on

23 of 24 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  6. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  7. 2013Auto-Encoding Variational Bayescited by 3,980
  8. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  9. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,006
  11. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
11 more
  1. 2022Video Diffusion Modelscited by 2,280
  2. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  3. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  4. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  5. 2022Tune-A-Video: One-Shot Tuning of Image Diffusion Models for Text-to-Video Generationcited by 958
  6. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 666
  7. 2023FateZero: Fusing Attentions for Zero-shot Text-based Video Editingcited by 462
  8. 2023Pix2Video: Video Editing using Image Diffusioncited by 318
  9. 2023Video-P2P: Video Editing with Cross-attention Controlcited by 312
  10. 2022EDICT: Exact Diffusion Inversion via Coupled Transformationscited by 256
  11. 2023Dreamix: Video Diffusion Models are General Video Editorscited by 189

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 10; 2025: 6; 2026: 1.

19 papers.

Sort:

By area:

  1. 2023Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioningcited by 254
  2. 2023A Survey on Video Diffusion Modelscited by 203
  3. 2024Diffusion Model-Based Video Editing: A Surveycited by 44
  4. 2024LeviTor: 3D Trajectory Oriented Image-to-Video Synthesiscited by 39
  5. 2024Audio-Synchronized Visual Animationcited by 32
  6. 2024Video Diffusion Models: A Surveycited by 28
  7. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generationcited by 23
  8. 2024AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generationcited by 21
  9. 2024QVD: Post-training Quantization for Video Diffusion Modelscited by 14
  10. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  11. 2025RFLAV: Rolling Flow matching for infinite Audio Video generationcited by 6
  12. 2025KeyVID: Keyframe-Aware Video Diffusion for Audio-Synchronized Visual Animationcited by 6
7 more
  1. 2025Syncphony: Synchronized Audio-to-Video Generation with Diffusion Transformerscited by 6
  2. 2024Artificial Intelligence for Biomedical Video Generationcited by 2
  3. 2025Scaling Up Audio-Synchronized Visual Animation: An Efficient Training Paradigmcited by 2
  4. 2025ProAV-DiT: A Projected Latent Diffusion Transformer for Efficient Synchronized Audio-Video Generationcited by 2
  5. 2024Fashion Style Editing with Generative Human Priorcited by 0
  6. 2024Combining Genre Classification and Harmonic-Percussive Features with Diffusion Models for Music-Video Generationcited by 0
  7. 2026TimeSteer: Inference-Time Speech Scheduling in Joint Audio-Visual Diffusion Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation
  2. 2026Diffusion Models for Joint Audio-Video Generation
  3. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
  4. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation
  5. 2024Text-to-Audio Generation Synchronized with Videos
  6. 2026Image-to-Video Diffusion: From Foundations to Open Frontiers
  7. 2023MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing
  8. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation