mlsanityData to 14 Oct 2026

TA-V2A: Textually Assisted Video-to-Audio Generation

arXiv 2503.10700 ↗First posted 12 Mar 2025cs.CV cs.MMcited by 5

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

As artificial intelligence-generated content (AIGC) continues to evolve, video-to-audio (V2A) generation has emerged as a key area with promising applications in multimedia editing, augmented reality, and automated content creation. While Transformer and Diffusion models have advanced audio generation, a significant challenge persists in extracting precise semantic information from videos, as current models often lose sequential context by relying solely on frame-based features. To address this, we present TA-V2A, a method that integrates language, audio, and video features to improve semantic representation in latent space. By incorporating large language models for enhanced video comprehension, our approach leverages text guidance to enrich semantic expression. Our diffusion model-based system utilizes automated text modulation to enhance inference quality and efficiency, providing personalized control through text-guided interfaces. This integration enhances semantic expression while ensuring temporal alignment, leading to more accurate and coherent video-to-audio generation.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 33 of its 35 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

33 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Text-to-Audio Generation Synchronized with Videos8 Mar
  2. 2024GLA-Grad: A Griffin-Lim Extended Waveform Generation Diffusion Model9 Feb
  3. 2023CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling8 Dec
  4. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models29 Jun
  5. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  6. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generation19 Dec

Other references · 27

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  4. 2022Classifier-Free Diffusion Guidancecited by 5,847
  5. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  6. 2018SlowFast Networks for Video Recognitioncited by 1,867
  7. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 771
  8. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  9. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  10. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  11. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  12. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
15 more
  1. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 574
  2. 2022AudioGen: Textually Guided Audio Generationcited by 295
  3. 2023Any-to-Any Generation via Composable Diffusioncited by 204
  4. 2021Taming Visually Guided Sound Generationcited by 136
  5. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  6. 2023Agent Attention: On the Integration of Softmax and Linear Attentioncited by 90
  7. 2023V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Modelscited by 65
  8. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  9. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  10. 2024Masked Generative Video-to-Audio Transformers with Enhanced Synchronicitycited by 28
  11. 2024Video-to-Audio Generation with Hidden Alignmentcited by 26
  12. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  13. 2024Correlation of Fr'echet Audio Distance With Human Perception of Environmental Audio Is Embedding Dependantcited by 15
  14. 2021Collaborative Learning to Generate Audio-Video Jointlycited by 6
  15. 2024PPPR: Portable Plug-in Prompt Refiner for Text to Audio Generationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 34
  2. 2025ViSAudio: End-to-End Video-Driven Binaural Spatial Audio Generationcited by 3
  3. 2026Towards Streaming Synchronized Spatial Audio Generation via Autoregressive Diffusion Transformercited by 1
  4. 2025MeanFlow-Accelerated Multimodal Video-to-Audio Synthesis via One-Step Generationcited by 0
  5. 2026AI-Based Sound Effect Generation: A Narrative Review of Generative Models Across Input Modalitiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024STA-V2A: Video-to-Audio Generation with Semantic and Temporal Alignment
  2. 2026Omni2Sound: Towards Unified Video-Text-to-Audio Generation
  3. 2024Video-to-Audio Generation with Fine-grained Temporal Semantics
  4. 2024TAVGBench: Benchmarking Text to Audible-Video Generation
  5. 2024Auffusion: Leveraging the Power of Diffusion and Large Language Models for Text-to-Audio Generation
  6. 2024Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model
  7. 2026AVGen-Bench: A Task-Driven Benchmark for Multi-Granular Evaluation of Text-to-Audio-Video Generation
  8. 2026LTX-2: Efficient Joint Audio-Visual Foundation Model