mlsanityData to 14 Oct 2026

Soundify: Matching Sound Effects to Video

arXiv 2112.09726 ↗First posted 17 Dec 2021cs.SD cs.CV cs.HCcited by 6

Not on the map. Why?

Abstract (arXiv)

In the art of video editing, sound helps add character to an object and immerse the viewer within a space. Through formative interviews with professional editors (N=10), we found that the task of adding sounds to video can be challenging. This paper presents Soundify, a system that assists editors in matching sounds to video. Given a video, Soundify identifies matching sounds, synchronizes the sounds to the video, and dynamically adjusts panning and volume to create spatial audio. In a human evaluation study (N=889), we show that Soundify is capable of matching sounds to video out-of-the-box for a diverse range of audio categories. In a within-subjects expert study (N=12), we demonstrate the usefulness of Soundify in helping video editors match sounds to video with lighter workload, reduced task completion time, and improved usability.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 23 of its 48 references that are in the map. First posted on arXiv 17 Dec 2021.

Builds on

23 of 48 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  4. 2016WaveNet: A Generative Model for Raw Audiocited by 1,722
  5. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  6. 2020Jukebox: A Generative Model for Musiccited by 542
  7. 2017Look, Listen and Learncited by 458
  8. 2019MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesiscited by 354
  9. 2018The Sound of Pixelscited by 259
  10. 2018Learning to Localize Sound Source in Visual Scenescited by 189
  11. 2016Grad-CAM: Why did you say that?cited by 161
  12. 2015Visually Indicated Soundscited by 128
11 more
  1. 2019GANSynth: Adversarial Neural Audio Synthesiscited by 116
  2. 2018Learning to Separate Object Sounds by Watching Unlabeled Videocited by 114
  3. 2019Co-Separating Sounds of Visual Objectscited by 110
  4. 2017Visual to Sound: Generating Natural Sound for Videos in the Wildcited by 98
  5. 2020Foley Music: Learning to Generate Music from Videoscited by 83
  6. 2017MidiNet: A Convolutional Generative Adversarial Network for Symbolic-domain Music Generationcited by 72
  7. 2020Generating Visually Aligned Sound from Videoscited by 67
  8. 2020Audeo: Audio Generation for a Silent Performance Videocited by 47
  9. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  10. 2020AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learningcited by 19
  11. 2021Dance2Music: Automatic Dance-driven Music Generationcited by 16

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 1.

6 papers.

  1. 2023ChatGPT is not all you need. A State of the Art Review of large Generative AI modelscited by 86
  2. 2023ChatGPT in the Age of Generative AI and Large Language Models: A Concise Surveycited by 10
  3. 2022VideoMap: Supporting Video Editing Exploration, Brainstorming, and Prototyping in the Latent Spacecited by 4
  4. 2023Bridging High-Quality Audio and Video via Language for Sound Effects Retrieval from Visual Queriescited by 2
  5. 2023What Do I Hear? Generating Sounds for Visuals with ChatGPTcited by 0
  6. 2024D&M: Enriching E-commerce Videos with Sound Effects by Key Moment Detection and SFX Matchingcited by 0