# Can Synthetic Audio From Generative Foundation Models Assist Audio Recognition and Speech Modeling?

arXiv: https://arxiv.org/abs/2406.08800 · HTML: https://mlsanity.com/paper/2406.08800/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 33 references that are in the map.
- First posted on arXiv 13 Jun 2024.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Diffusion Models in Vision: A Survey](https://mlsanity.com/paper/2209.04747/) (2022)
- [Speech Commands: A Dataset for Limited-Vocabulary Speech Recognition](https://mlsanity.com/paper/1804.03209/) (2018)
- [VGGSound: A Large-scale Audio-Visual Dataset](https://mlsanity.com/paper/2004.14368/) (2020)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [FSD50K: An Open Dataset of Human-Labeled Sound Events](https://mlsanity.com/paper/2010.00475/) (2020)
- [Is synthetic data from generative models ready for image recognition?](https://mlsanity.com/paper/2210.07574/) (2022)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Masked Autoencoders that Listen](https://mlsanity.com/paper/2207.06405/) (2022)
- [SSAST: Self-Supervised Audio Spectrogram Transformer](https://mlsanity.com/paper/2110.09784/) (2021)
- [Diversity is Definitely Needed: Improving Model-Agnostic Zero-shot Classification via Stable Diffusion](https://mlsanity.com/paper/2302.03298/) (2023)
- [SynthASR: Unlocking Synthetic Data for Speech Recognition](https://mlsanity.com/paper/2106.07803/) (2021)
- [A Data-Centric Approach for Training Deep Neural Networks with Less Data](https://mlsanity.com/paper/2110.03613/) (2021)
- [GPT-FL: Generative Pre-trained Model-Assisted Federated Learning](https://mlsanity.com/paper/2306.02210/) (2023)
- [Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmenting](https://mlsanity.com/paper/2309.08108/) (2023)
- [Is Synthetic Image Useful for Transfer Learning? An Investigation into Data Generation, Volume, and Utilization](https://mlsanity.com/paper/2403.19866/) (2024)
- [Can Text-to-image Model Assist Multi-modal Learning for Visual Recognition with Visual Modality Missing?](https://mlsanity.com/paper/2402.09036/) (2024)

## Built on by (most cited first, 4 of 4)

- [Synthio: Augmenting Small-Scale Audio Classification Datasets with Synthetic Data](https://mlsanity.com/paper/2410.02056/) (2024)
- [MambaFoley: Foley Sound Generation using Selective State-Space Models](https://mlsanity.com/paper/2409.09162/) (2024)
- [Mind the Prompt: Prompting Strategies in Audio Generations for Improving Sound Classification](https://mlsanity.com/paper/2504.03329/) (2025)
- [SpectCount: Spectrotemporal Counting via Synthetic Signals Improves Large Audio Language Models](https://mlsanity.com/paper/2606.06907/) (2026)

## Abstract (arXiv)

Recent advances in foundation models have enabled audio-generative models that produce high-fidelity sounds associated with music, events, and human actions. Despite the success achieved in modern audio-generative models, the conventional approach to assessing the quality of the audio generation relies heavily on distance metrics like Frechet Audio Distance. In contrast, we aim to evaluate the quality of audio generation by examining the effectiveness of using them as training data. Specifically, we conduct studies to explore the use of synthetic audio for audio recognition. Moreover, we investigate whether synthetic audio can serve as a resource for data augmentation in speech-related modeling. Our comprehensive experiments demonstrate the potential of using synthetic audio for audio recognition and speech-related modeling. Our code is available at https://github.com/usc-sail/SynthAudio.