# AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning

arXiv: https://arxiv.org/abs/2002.10981 · HTML: https://mlsanity.com/paper/2002.10981/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Feb 2020.

## Built on by (most cited first, 19 of 19)

- [V2A-Mapper: A Lightweight Solution for Vision-to-Audio Generation by Connecting Foundation Models](https://mlsanity.com/paper/2308.09300/) (2023)
- [Conditional Generation of Audio from Video via Foley Analogies](https://mlsanity.com/paper/2304.08490/) (2023)
- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis](https://mlsanity.com/paper/2401.09294/) (2024)
- [A Simple but Strong Baseline for Sounding Video Generation: Effective Adaptation of Audio and Video Diffusion Models for Joint Generation](https://mlsanity.com/paper/2409.17550/) (2024)
- [It's Time for Artistic Correspondence in Music and Video](https://mlsanity.com/paper/2206.07148/) (2022)
- [Sound Localization from Motion: Jointly Learning Sound Direction and Camera Rotation](https://mlsanity.com/paper/2303.11329/) (2023)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [Leveraging Pre-trained AudioLDM for Sound Generation: A Benchmark Study](https://mlsanity.com/paper/2303.03857/) (2023)
- [A Proposal for Foley Sound Synthesis Challenge](https://mlsanity.com/paper/2207.10760/) (2022)
- [Soundify: Matching Sound Effects to Video](https://mlsanity.com/paper/2112.09726/) (2021)
- [FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignment](https://mlsanity.com/paper/2412.15023/) (2024)
- [A Demand-Driven Perspective on Generative Audio AI](https://mlsanity.com/paper/2307.04292/) (2023)
- [Human Activity Recognition Using Cascaded Dual Attention CNN and Bi-Directional GRU Framework](https://mlsanity.com/paper/2208.05034/) (2022)
- [Improving CNN-base Stock Trading By Considering Data Heterogeneity and Burst](https://mlsanity.com/paper/2303.09407/) (2023)
- [What Do I Hear? Generating Sounds for Visuals with ChatGPT](https://mlsanity.com/paper/2311.05609/) (2023)
- [D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching](https://mlsanity.com/paper/2408.13226/) (2024)
- [StereoSync: Spatially-Aware Stereo Audio Generation from Video](https://mlsanity.com/paper/2510.05828/) (2025)
- [AMAVA: Adaptive Motion-Aware Video-to-Audio Framework for Visually-Impaired Assistance](https://mlsanity.com/paper/2604.23909/) (2026)

## Abstract (arXiv)

In movie productions, the Foley Artist is responsible for creating an overlay soundtrack that helps the movie come alive for the audience. This requires the artist to first identify the sounds that will enhance the experience for the listener thereby reinforcing the Directors's intention for a given scene. In this paper, we present AutoFoley, a fully-automated deep learning tool that can be used to synthesize a representative audio track for videos. AutoFoley can be used in the applications where there is either no corresponding audio file associated with the video or in cases where there is a need to identify critical scenarios and provide a synthesized, reinforced soundtrack. An important performance criterion of the synthesized soundtrack is to be time-synchronized with the input video, which provides for a realistic and believable portrayal of the synthesized sound. Unlike existing sound prediction and generation architectures, our algorithm is capable of precise recognition of actions as well as inter-frame relations in fast moving video clips by incorporating an interpolation technique and Temporal Relationship Networks (TRN). We employ a robust multi-scale Recurrent Neural Network (RNN) associated with a Convolutional Neural Network (CNN) for a better understanding of the intricate input-to-output associations over time. To evaluate AutoFoley, we create and introduce a large scale audio-video dataset containing a variety of sounds frequently used as Foley effects in movies. Our experiments show that the synthesized sounds are realistically portrayed with accurate temporal synchronization of the associated visual inputs. Human qualitative testing of AutoFoley show over 73% of the test subjects considered the generated soundtrack as original, which is a noteworthy improvement in cross-modal research in sound synthesis.