mlsanityData to 14 Oct 2026

TARO: Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning for Synchronized Video-to-Audio Synthesis

arXiv 2504.05684 ↗First posted 8 Apr 2025cs.SD cs.AI cs.CVcited by 10

Not on the map. Why?

Abstract (arXiv)

This paper introduces Timestep-Adaptive Representation Alignment with Onset-Aware Conditioning (TARO), a novel framework for high-fidelity and temporally coherent video-to-audio synthesis. Built upon flow-based transformers, which offer stable training and continuous transformations for enhanced synchronization and audio quality, TARO introduces two key innovations: (1) Timestep-Adaptive Representation Alignment (TRA), which dynamically aligns latent representations by adjusting alignment strength based on the noise schedule, ensuring smooth evolution and improved fidelity, and (2) Onset-Aware Conditioning (OAC), which integrates onset cues that serve as sharp event-driven markers of audio-relevant visual moments to enhance synchronization with dynamic visual events. Extensive experiments on the VGGSound and Landscape datasets demonstrate that TARO outperforms prior methods, achieving relatively 53% lower Frechet Distance (FD), 29% lower Frechet Audio Distance (FAD), and a 97.19% Alignment Accuracy, highlighting its superior audio quality and synchronization precision.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 8 Apr 2025.

Builds on

38 of 47 references.

References · 38

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  5. 2022Classifier-Free Diffusion Guidancecited by 5,847
  6. 2022Flow Matching for Generative Modelingcited by 5,702
  7. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  8. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  9. 2024CogVideoX: Text-to-Video Diffusion Models with An Expert Transformercited by 2,261
  10. 2024SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformerscited by 737
  11. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 719
  12. 2024Representation Alignment for Generation: Training Diffusion Transformers Is Easier Than You Thinkcited by 703
26 more
  1. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 648
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 388
  3. 2023Voicebox: Text-Guided Multilingual Universal Speech Generation at Scalecited by 379
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 344
  5. 2022MM-Diffusion: Learning Multi-Modal Diffusion Models for Joint Audio and Video Generationcited by 227
  6. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  7. 2021Taming Visually Guided Sound Generationcited by 136
  8. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Modelscited by 134
  9. 2023Audiobox: Unified Audio Generation with Natural Language Promptscited by 120
  10. 2024Seeing and Hearing: Open-domain Visual-Audio Generation with Diffusion Latent Alignerscited by 111
  11. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 110
  12. 2022I Hear Your True Colors: Image Guided Audio Generationcited by 92
  13. 2020Generating Visually Aligned Sound from Videoscited by 67
  14. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 53
  15. 2024Frieren: Efficient Video-to-Audio Generation Network with Rectified Flow Matchingcited by 53
  16. 2023VoiceFlow: Efficient Text-to-Speech with Rectified Flow Matchingcited by 49
  17. 2024Read, Watch and Scream! Sound Generation from Text and Videocited by 48
  18. 2022Sound-Guided Semantic Video Generationcited by 35
  19. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesiscited by 24
  20. 2024FRAG: Frequency Adapting Group for Diffusion Video Editingcited by 15
  21. 2024DNI: Dilutional Noise Initialization for Diffusion Video Editingcited by 12
  22. 2024TPC: Test-time Procrustes Calibration for Diffusion-based Human Image Animationcited by 12
  23. 2025ITA-MDT: Image-Timestep-Adaptive Masked Diffusion Transformer Framework for Image-Based Virtual Try-Oncited by 10
  24. 2024MDSGen: Fast and Efficient Masked Diffusion Temporal-Aware Transformers for Open-Domain Sound Generationcited by 9
  25. 2022A Proposal for Foley Sound Synthesis Challengecited by 8
  26. 2024Predictive Coding for Decision Transformercited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 6.

10 papers.

  1. 2025Policy Learning from Large Vision-Language Model Feedback without Reward Modelingcited by 4
  2. 2025Occlusion-robust Stylization for Drawing-based 3D Animationcited by 3
  3. 2025PAVAS: Physics-Aware Video-to-Audio Synthesiscited by 2
  4. 2026Test-Time Conditioning with Representation-Aligned Visual Featurescited by 2
  5. 2026Fast and Highly Expressive Policy Learning for Offline Reinforcement Learning via Bootstrapped Flow Q-Learningcited by 2
  6. 2026AG-REPA: Causal Layer Selection for Representation Alignment in Audio Flow Matchingcited by 1
  7. 2026AirSplat: Alignment and Rating for Robust Feed-Forward 3D Gaussian Splattingcited by 1
  8. 2025SALSA-V: Shortcut-Augmented Long-form Synchronized Audio from Videoscited by 0
  9. 2026CMA-OT: Hierarchical Expert Supervision for Dance-to-Music Generationcited by 0
  10. 2026Joint and Cross-Modal Video-Audio Generation and Editing: A Unified Formulation and Design Taxonomycited by 0