mlsanityData to 7 Oct 2026

T-FOLEY: A Controllable Waveform-Domain Diffusion Model for Temporal-Event-Guided Foley Sound Synthesis

arXiv 2401.09294 ↗First posted 17 Jan 2024cs.SD cs.AI cs.LGcited by 17

On the map: Diffusion models → Audio, speech and music

Abstract (arXiv)

Foley sound, audio content inserted synchronously with videos, plays a critical role in the user experience of multimedia content. Recently, there has been active research in Foley sound synthesis, leveraging the advancements in deep generative models. However, such works mainly focus on replicating a single sound class or a textual sound description, neglecting temporal information, which is crucial in the practical applications of Foley sound. We present T-Foley, a Temporal-event-guided waveform generation model for Foley sound synthesis. T-Foley generates high-quality audio using two conditions: the sound class and temporal event feature. For temporal conditioning, we devise a temporal event feature and a novel conditioning technique named Block-FiLM. T-Foley achieves superior performance in both objective and subjective evaluation metrics and generates Foley sound well-synchronized with the temporal events. Additionally, we showcase T-Foley's practical applications, particularly in scenarios involving vocal mimicry for temporal event control. We show the demo on our companion website.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Placed in Audio, speech and music, one of 14 directions in Diffusion models. Builds on 22 of its 24 references that are in the map. First posted on arXiv 17 Jan 2024.

Builds on

22 of 24 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Models29 Jan
  2. 2022Diffsound: Discrete Diffusion Model for Text-to-sound Generation20 Jul
  3. 2021CRASH: Raw Audio Score-based Generative Modeling for Controllable High-resolution Drum Sound Synthesis14 Jun

Other references · 19

  1. 2022Classifier-Free Diffusion Guidancecited by 5,717
  2. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  3. 2021Variational Diffusion Modelscited by 1,145
  4. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  5. 2020WaveGrad: Estimating Gradients for Waveform Generationcited by 620
  6. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  7. 2022AudioGen: Textually Guided Audio Generationcited by 292
  8. 2020Generating Visually Aligned Sound from Videoscited by 67
  9. 2023Conditional Generation of Audio from Video via Foley Analogiescited by 52
  10. 2019Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulationscited by 36
  11. 2020DrumGAN: Synthesis of Drum Sounds With Timbral Feature Conditioning Using Generative Adversarial Networkscited by 34
  12. 2022Full-band General Audio Synthesis with Score-based Diffusioncited by 32
7 more
  1. 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
  2. 2021FoleyGAN: Visually Guided Generative Adversarial Network-Based Synchronous Sound Generation in Silent Videoscited by 20
  3. 2020AutoFoley: Artificial Synthesis of Synchronized Sound Tracks for Silent Videos with Deep Learningcited by 19
  4. 2023Foley Sound Synthesis at the DCASE 2023 Challengecited by 19
  5. 2022VarietySound: Timbre-Controllable Video to Sound Generation via Unsupervised Information Disentanglementcited by 11
  6. 2021Generating Diverse Realistic Laughter for Interactive Artcited by 5
  7. 2021Neural Synthesis of Footsteps Sound Effects with Generative Adversarial Networkscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 6; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  2. 2024FoleyCrafter: Bring Silent Videos to Life with Lifelike and Synchronized Soundscited by 109
  3. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  4. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  5. 2024Video-Foley: Two-Stage Video-To-Sound Generation via Temporal Event Condition For Foley Soundcited by 16
  6. 2024Naturalistic Music Decoding from EEG Data via Latent Diffusion Modelscited by 8
  7. 2026JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusioncited by 6
  8. 2024The Sound of Water: Inferring Physical Properties from Pouring Liquidscited by 5
  9. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  10. 2024MambaFoley: Foley Sound Generation using Selective State-Space Modelscited by 3
  11. 2025Hear What Matters! Text-conditioned Selective Video-to-Audio Generationcited by 3
  12. 2025When Humans Growl and Birds Speak: High-Fidelity Voice Conversion from Human to Animal and Designed Soundscited by 1
5 more
  1. 2025Audio Palette: A Diffusion Transformer with Multi-Signal Conditioning for Controllable Foley Synthesiscited by 1
  2. 2025StereoSync: Spatially-Aware Stereo Audio Generation from Videocited by 0
  3. 2025FoleyGRAM: Video-to-Audio Generation with GRAM-Aligned Multimodal Encoderscited by 0
  4. 2026A Production-Oriented Framework for Evaluation of SFX Generationcited by 0
  5. 2026Exploring Efficient Waveform Diffusion Models for Foley Sound Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Text-Driven Foley Sound Generation With Latent Diffusion Model
  2. 2026FoleyGenEx: Unified Video-to-Audio Generation with Multi-Modal Control, Temporal Alignment, and Semantic Precision
  3. 2024Latent CLAP Loss for Better Foley Sound Synthesis
  4. 2023Latent Diffusion Model Based Foley Sound Generation System For DCASE Challenge 2023 Task 7
  5. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generation
  6. 2023SyncFusion: Multimodal Onset-synchronized Video-to-Audio Foley Synthesis
  7. 2023Diff-Foley: Synchronized Video-to-Audio Synthesis with Latent Diffusion Models
  8. 2025Kling-Foley: Multimodal Diffusion Transformer for High-Quality Video-to-Audio Generation