mlsanityData to 7 Oct 2026

FALL-E: A Foley Sound Synthesis Model and Strategies

arXiv 2306.09807 ↗First posted 16 Jun 2023eess.AS cs.LG cs.SDcited by 2

Not on the map. Why?

Abstract (arXiv)

This paper introduces FALL-E, a foley synthesis system and its training/inference strategies. The FALL-E model employs a cascaded approach comprising low-resolution spectrogram generation, spectrogram super-resolution, and a vocoder. We trained every sound-related model from scratch using our extensive datasets, and utilized a pre-trained language model. We conditioned the model with dataset-specific texts, enabling it to learn sound quality and recording environment based on text input. Moreover, we leveraged external language models to improve text descriptions of our datasets and performed prompt engineering for quality, coherence, and diversity. FALL-E was evaluated by an objective measure as well as listening tests in the DCASE 2023 challenge Task 7. The submission achieved the second place on average, while achieving the best score for diversity, second place for audio quality, and third place for class fitness.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 25 of its 29 references that are in the map. First posted on arXiv 16 Jun 2023.

Builds on

25 of 29 references.

References · 25

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  5. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  6. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  7. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  8. 2021Cascaded Diffusion Models for High Fidelity Image Generationcited by 1,155
  9. 2020HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesiscited by 1,139
  10. 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
  11. 2020FastSpeech 2: Fast and High-Quality End-to-End Text to Speechcited by 710
  12. 2022Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentationcited by 707
13 more
  1. 2022High Fidelity Neural Audio Compressioncited by 673
  2. 2023A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPTcited by 612
  3. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  4. 2023AudioLDM: Text-to-Audio Generation with Latent Diffusion Modelscited by 592
  5. 2019Clotho: An Audio Captioning Datasetcited by 354
  6. 2022AudioGen: Textually Guided Audio Generationcited by 292
  7. 2022BigVGAN: A Universal Neural Vocoder with Large-Scale Trainingcited by 245
  8. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  9. 2018Towards End-to-End Prosody Transfer for Expressive Speech Synthesis with Tacotroncited by 171
  10. 2021Symbolic Music Generation with Diffusion Modelscited by 152
  11. 2023Foley Sound Synthesis at the DCASE 2023 Challengecited by 19
  12. 2022A Proposal for Foley Sound Synthesis Challengecited by 8
  13. 2019High-quality Speech Synthesis Using Super-resolution Mel-Spectrogramcited by 7

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1.

2 papers.

  1. 2023A Demand-Driven Perspective on Generative Audio AIcited by 4
  2. 2024MambaFoley: Foley Sound Generation using Selective State-Space Modelscited by 3