mlsanityData to 7 Oct 2026

AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation

arXiv 2412.15191 ↗First posted 19 Dec 2024cs.CV cs.LG cs.SDcited by 21

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

We propose AV-Link, a unified framework for Video-to-Audio (A2V) and Audio-to-Video (A2V) generation that leverages the activations of frozen video and audio diffusion models for temporally-aligned cross-modal conditioning. The key to our framework is a Fusion Block that facilitates bidirectional information exchange between video and audio diffusion models through temporally-aligned self attention operations. Unlike prior work that uses dedicated models for A2V and V2A tasks and relies on pretrained feature extractors, AV-Link achieves both tasks in a single self-contained framework, directly leveraging features obtained by the complementary modality (i.e. video features to generate audio, or audio features to generate video). Extensive automatic and subjective evaluations demonstrate that our method achieves a substantial improvement in audio-video synchronization, outperforming more expensive baselines such as the MovieGen video-to-audio model.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 88 of its 109 references that are in the map. First posted on arXiv 19 Dec 2024.

Builds on

88 of 109 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation9 Oct
  2. 2024MM-LDM: Multi-Modal Latent Diffusion Model for Sounding Video Generation2 Oct
  3. 2024A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation26 Sep
  4. 2024LoVA: Long-form Video-to-Audio Generation23 Sep
  5. 2024AV-DiT: Efficient Audio-Visual Diffusion Transformer for Joint Audio and Video Generation11 Jun
  6. 2024MMDisCo: Multi-Modal Discriminator-Guided Cooperative Diffusion for Joint Audio and Video Generation28 May
  7. 2024A Versatile Diffusion Transformer with Mixture of Noise Levels for Audiovisual Generation22 May
  8. 2024SonicDiffusion: Audio-Driven Image Generation and Editing with Pretrained Diffusion Models1 May
  9. 2024Tango 2: Aligning Diffusion-based Text-to-Audio Generations through Direct Preference Optimization15 Apr
  10. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation2 Jan
  11. 2023Audiobox: Unified Audio Generation with Natural Language Prompts25 Dec
  12. 2023CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling8 Dec
7 more
  1. 2023The Power of Sound (TPoS): Audio Reactive Video Generation with Stable Diffusion8 Sep
  2. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining10 Aug
  3. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models29 Jun
  4. 2023Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation29 May
  5. 2023AADiff: Audio-Aligned Video Synthesis with Text-to-Image Diffusion6 May
  6. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  7. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation19 Dec

Other references · 69

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2022Scalable Diffusion Models with Transformerscited by 5,931
  4. 2022Classifier-Free Diffusion Guidancecited by 5,717
  5. 2022Flow Matching for Generative Modelingcited by 5,426
  6. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,094
  8. 2022Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flowcited by 3,550
  9. 2016Improved Techniques for Training GANscited by 2,938
  10. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,526
  11. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  12. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,198
38 more
  1. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,725
  2. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,465
  3. 2018Towards Accurate Generative Models of Video: A New Metric & Challengescited by 1,204
  4. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  5. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  6. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 867
  7. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
  8. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  9. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  10. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 633
  11. 2024Movie Gen: A Cast of Media Foundation Modelscited by 614
  12. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  13. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 475
  14. 2024Panda-70M: Captioning 70M Videos with Multiple Cross-Modality Teacherscited by 389
  15. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  16. 2024Pyramidal Flow Matching for Efficient Video Generative Modelingcited by 354
  17. 2023VideoFusion: Decomposed Diffusion Models for High-Quality Video Generationcited by 330
  18. 2023Diffusion Hyperfeatures: Searching Through Time and Space for Semantic Correspondencecited by 211
  19. 2016FMA: A Dataset For Music Analysiscited by 205
  20. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  21. 2021Taming Visually Guided Sound Generationcited by 136
  22. 2023Unsupervised Semantic Correspondence Using Stable Diffusioncited by 120
  23. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  24. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  25. 2024Snap Video: Scaled Spatiotemporal Transformers for Text-to-Video Synthesiscited by 105
  26. 2023Diffusion Model as Representation Learnercited by 95
  27. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  28. 2023Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptationcited by 68
  29. 2023The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluationcited by 68
  30. 2024Allegro: Open the Black Box of Commercial-Level Video Generation Modelcited by 68
  31. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  32. 2024T2V-Turbo-v2: Enhancing Video Generation Model Post-Training through Data, Reward, and Conditional Guidance Designcited by 54
  33. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  34. 2024Temporally Aligned Audio for Video with Autoregressioncited by 51
  35. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  36. 2023Epic-Sounds: A Large-scale Dataset of Actions That Soundcited by 47
  37. 2023FoleyGen: Visually-Guided Audio Generationcited by 40
  38. 2022Sound-Guided Semantic Video Generationcited by 35

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025Controllable Video Generation: A Surveycited by 93
  2. 2025UniAVGen: Unified Audio and Video Generation with Asymmetric Cross-Modal Interactionscited by 35
  3. 2026OmniRoam: World Wandering via Long-Horizon Panoramic Video Generationcited by 15
  4. 2025Improving Progressive Generation with Decomposable Flow Matchingcited by 6
  5. 2026AVControl: Efficient Framework for Training Audio-Visual Controlscited by 5
  6. 2025MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generationcited by 4
  7. 2026One Model, Many Budgets: Elastic Latent Interfaces for Diffusion Transformerscited by 3
  8. 2026AVBench: Human-Aligned and Automated Evaluation Benchmark for Audio-Video Generative Modelscited by 3
  9. 2026Baton: Explicit Semantic Blueprints for Joint Video-Audio Generationcited by 3
  10. 2026Composing Driving Worlds through Disentangled Control for Adversarial Scenario Generationcited by 2
  11. 2025Generative Augmented Reality: Paradigms, Technologies, and Future Applicationscited by 1
  12. 2026Mutual Forcing: Dual-Mode Self-Evolution for Fast Autoregressive Audio-Video Character Generationcited by 1
9 more
  1. 2026Benchmarking Single-Factor Physical Video-to-Audio Generationcited by 1
  2. 2026Efficient Audio-Visual Generation via Synchrony-Aware Cross-Modal Sparse Attentioncited by 1
  3. 2026The Attention Triangle in Audio-Video Modelscited by 1
  4. 2025Evaluating Text-to-Image and Text-to-Video Synthesis with a Conditional Fr'echet Distancecited by 0
  5. 2025Model-Guided Dual-Role Alignment for High-Fidelity Open-Domain Video-to-Audio Generationcited by 0
  6. 2025JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editingcited by 0
  7. 2026MAVIN: Multi-Shot Audio-Visual Generation with Customized Narrative Controlcited by 0
  8. 2026The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generationcited by 0
  9. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Diffusion Models for Joint Audio-Video Generation
  2. 20253MDiT: Unified Tri-Modal Diffusion Transformer for Text-Driven Synchronized Audio-Video Generation
  3. 2026HarmoniDPO: Video-guided Audio Generation via Preference-Optimized Diffusion
  4. 2026AV-GRPO: Modality-Anchored Decoupling Diffusion Reinforcement Learning for Joint Audio-Video Generation
  5. 2026Improving Joint Audio-Video Generation with Cross-Modal Context Learning
  6. 2025Training-Free Multimodal Guidance for Video to Audio Generation
  7. 2026OmniForcing: Unleashing Real-time Joint Audio-Visual Generation
  8. 2026Talker-T2AV: Joint Talking Audio-Video Generation with Autoregressive Diffusion Modeling