# Soundify: Matching Sound Effects to Video

arXiv: https://arxiv.org/abs/2112.09726 · HTML: https://mlsanity.com/paper/2112.09726/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 48 references that are in the map.
- First posted on arXiv 17 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization](https://mlsanity.com/paper/1610.02391/) (2016)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [DiffWave: A Versatile Diffusion Model for Audio Synthesis](https://mlsanity.com/paper/2009.09761/) (2020)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [The Sound of Pixels](https://mlsanity.com/paper/1804.03160/) (2018)
- [Learning to Localize Sound Source in Visual Scenes](https://mlsanity.com/paper/1803.03849/) (2018)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [Visually Indicated Sounds](https://mlsanity.com/paper/1512.08512/) (2015)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [Learning to Separate Object Sounds by Watching Unlabeled Video](https://mlsanity.com/paper/1804.01665/) (2018)
- [Co-Separating Sounds of Visual Objects](https://mlsanity.com/paper/1904.07750/) (2019)
- [Visual to Sound: Generating Natural Sound for Videos in the Wild](https://mlsanity.com/paper/1712.01393/) (2017)
- [Foley Music: Learning to Generate Music from Videos](https://mlsanity.com/paper/2007.10984/) (2020)
- [MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generation](https://mlsanity.com/paper/1703.10847/) (2017)
- [Generating Visually Aligned Sound from Videos](https://mlsanity.com/paper/2008.00820/) (2020)
- [Audeo: Audio Generation for a Silent Performance Video](https://mlsanity.com/paper/2006.14348/) (2020)
- [FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videos](https://mlsanity.com/paper/2107.09262/) (2021)
- [AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learning](https://mlsanity.com/paper/2002.10981/) (2020)
- [Dance2Music: Automatic Dance-driven Music Generation](https://mlsanity.com/paper/2107.06252/) (2021)

## Built on by (most cited first, 6 of 6)

- [ChatGPT is not all you need. A State of the Art Review of large Generative AI models](https://mlsanity.com/paper/2301.04655/) (2023)
- [ChatGPT in the Age of Generative AI and Large Language Models: A Concise Survey](https://mlsanity.com/paper/2307.04251/) (2023)
- [VideoMap: Supporting Video Editing Exploration, Brainstorming, and Prototyping in the Latent Space](https://mlsanity.com/paper/2211.12492/) (2022)
- [Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queries](https://mlsanity.com/paper/2308.09089/) (2023)
- [What Do I Hear? Generating Sounds for Visuals with ChatGPT](https://mlsanity.com/paper/2311.05609/) (2023)
- [D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matching](https://mlsanity.com/paper/2408.13226/) (2024)

## Abstract (arXiv)

In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.