VIFS: An End-to-End Variational Inference for Foley Sound Synthesis
Not on the map. Why?
Abstract (arXiv)
The goal of DCASE 2023 Challenge Task 7 is to generate various sound clips for Foley sound synthesis (FSS) by "category-to-sound" approach. "Category" is expressed by a single index while corresponding "sound" covers diverse and different sound examples. To generate diverse sounds for a given category, we adopt VITS, a text-to-speech (TTS) model with variational inference. In addition, we apply various techniques from speech synthesis including PhaseAug and Avocodo. Different from TTS models which generate short pronunciation from phonemes and speaker identity, the category-to-sound problem requires generating diverse sounds just from a category index. To compensate for the difference while maintaining consistency within each audio clip, we heavily modified the prior encoder to enhance consistency with posterior latent variables. This introduced additional Gaussian on the prior encoder which promotes variance within the category. With these modifications, we propose VIFS, variational inference for end-to-end Foley sound synthesis, which generates diverse high-quality sounds.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 20 of its 22 references that are in the map. First posted on arXiv 8 Jun 2023.
Builds on
20 of 22 references.
References · 20
- 2022Generative Adversarial Networkscited by 5,603
- 2013Auto-Encoding Variational Bayescited by 3,980
- 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
- 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
- 2021Conditional Variational Autoencoder with Adversarial Learning for End-to-End Text-to-Speechcited by 555
- 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
- 2022AudioGen: Textually Guided Audio Generationcited by 292
- 2019Residual Flows for Invertible Generative Modelingcited by 256
- 2018Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithmscited by 221
- 2018Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition Systemcited by 45
- 2021VISinger: Variational Inference with Adversarial Learning for End-to-End Singing Voice Synthesiscited by 38
- 2021Conditional Sound Generation Using Neural Discrete Time-Frequency Representation Learningcited by 24
8 more
- 2022Frequency Dynamic Convolution: Frequency-Adaptive Pattern Recognition for Sound Event Detectioncited by 23
- 2023Foley Sound Synthesis at the DCASE 2023 Challengecited by 19
- 2022Avocodo: Generative Adversarial Network for Artifact-free Vocodercited by 18
- 2021FilterAugment: An Acoustic Environmental Data Augmentation Methodcited by 8
- 2022A Proposal for Foley Sound Synthesis Challengecited by 8
- 2022PhaseAug: A Differentiable Augmentation for Speech Synthesis to Simulate One-to-Many Mappingcited by 8
- 2021Heavily Augmented Sound Event Detection utilizing Weak Predictionscited by 6
- 2023PITS: Variational Pitch Inference without Fundamental Frequency for End-to-End Pitch-controllable TTScited by 4
Built on by
Citing papers per year: 2025: 2.
2 papers.