# Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis

arXiv: https://arxiv.org/abs/2409.08628 · HTML: https://mlsanity.com/paper/2409.08628/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2024.

## Built on by (most cited first, 5 of 5)

- [MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis](https://mlsanity.com/paper/2412.15322/) (2024)
- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [AV-Link: Temporally-Aligned Diffusion Features for Cross-Modal Audio-Video Generation](https://mlsanity.com/paper/2412.15191/) (2024)
- [DeepSound-V1: Start to Think Step-by-Step in the Audio Generation from Videos](https://mlsanity.com/paper/2503.22208/) (2025)
- [Exploring Efficient Waveform Diffusion Models for Foley Sound Generation](https://mlsanity.com/paper/2607.29148/) (2026)

## Abstract (arXiv)

Our research introduces an innovative framework for video-to-audio synthesis, which solves the problems of audio-video desynchronization and semantic loss in the audio. By incorporating a semantic alignment adapter and a temporal synchronization adapter, our method significantly improves semantic integrity and the precision of beat point synchronization, particularly in fast-paced action sequences. Utilizing a contrastive audio-visual pre-trained encoder, our model is trained with video and high-quality audio data, improving the quality of the generated audio. This dual-adapter approach empowers users with enhanced control over audio semantics and beat effects, allowing the adjustment of the controller to achieve better results. Extensive experiments substantiate the effectiveness of our framework in achieving seamless audio-visual alignment.