# Catch-A-Waveform: Learning to Generate Audio from a Single Short Example

arXiv: https://arxiv.org/abs/2106.06426 · HTML: https://mlsanity.com/paper/2106.06426/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 46 of its 64 references that are in the map.
- First posted on arXiv 11 Jun 2021.

## Builds on (most cited first, 30 of 46)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Multi-Scale Context Aggregation by Dilated Convolutions](https://mlsanity.com/paper/1511.07122/) (2015)
- [Conditional Image Generation with PixelCNN Decoders](https://mlsanity.com/paper/1606.05328/) (2016)
- [Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networks](https://mlsanity.com/paper/1602.07868/) (2016)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [SinGAN: Learning a Generative Model from a Single Natural Image](https://mlsanity.com/paper/1905.01164/) (2019)
- [WaveGlow: A Flow-based Generative Network for Speech Synthesis](https://mlsanity.com/paper/1811.00002/) (2018)
- [Glow-TTS: A Generative Flow for Text-to-Speech via Monotonic Alignment Search](https://mlsanity.com/paper/2005.11129/) (2020)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Parallel WaveNet: Fast High-Fidelity Speech Synthesis](https://mlsanity.com/paper/1711.10433/) (2017)
- [FMA: A Dataset For Music Analysis](https://mlsanity.com/paper/1612.01840/) (2016)
- [Precomputed Real-Time Texture Synthesis with Markovian Generative Adversarial Networks](https://mlsanity.com/paper/1604.04382/) (2016)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [SEGAN: Speech Enhancement Generative Adversarial Network](https://mlsanity.com/paper/1703.09452/) (2017)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [Neural Voice Cloning with a Few Samples](https://mlsanity.com/paper/1802.06006/) (2018)
- [High Fidelity Speech Synthesis with Adversarial Networks](https://mlsanity.com/paper/1909.11646/) (2019)
- [ClariNet: Parallel Wave Generation in End-to-End Text-to-Speech](https://mlsanity.com/paper/1807.07281/) (2018)
- [Positional Encoding as Spatial Inductive Bias in GANs](https://mlsanity.com/paper/2012.05217/) (2020)
- [FloWaveNet : A Generative Flow for Raw Audio](https://mlsanity.com/paper/1811.02155/) (2018)
- [WaveFlow: A Compact Flow-based Model for Raw Audio](https://mlsanity.com/paper/1912.01219/) (2019)
- [Deep Geometric Texture Synthesis](https://mlsanity.com/paper/2007.00074/) (2020)
- [AGAIN-VC: A One-shot Voice Conversion using Activation Guidance and Adaptive Instance Normalization](https://mlsanity.com/paper/2011.00316/) (2020)
- [Hierarchical Patch VAE-GAN: Generating Diverse Videos from a Single Sample](https://mlsanity.com/paper/2006.12226/) (2020)
- [Temporal FiLM: Capturing Long-Range Sequence Dependencies with Feature-Wise Modulations](https://mlsanity.com/paper/1909.06628/) (2019)
- [Fast Spectrogram Inversion using Multi-head Convolutional Neural Networks](https://mlsanity.com/paper/1808.06719/) (2018)

## Built on by (most cited first, 15 of 15)

- [Taming Visually Guided Sound Generation](https://mlsanity.com/paper/2110.08791/) (2021)
- [Solving Audio Inverse Problems with a Diffusion Model](https://mlsanity.com/paper/2210.15228/) (2022)
- [SinDDM: A Single Image Denoising Diffusion Model](https://mlsanity.com/paper/2211.16582/) (2022)
- [Multi-instrument Music Synthesis with Spectrogram Diffusion](https://mlsanity.com/paper/2206.05408/) (2022)
- [Learning to Generate 3D Shapes from a Single Example](https://mlsanity.com/paper/2208.02946/) (2022)
- [Sin3DM: Learning a Diffusion Model from a Single 3D Textured Shape](https://mlsanity.com/paper/2305.15399/) (2023)
- [Dilated convolution with learnable spacings](https://mlsanity.com/paper/2112.03740/) (2021)
- [The Ethical Implications of Generative Audio Models: A Systematic Literature Review](https://mlsanity.com/paper/2307.05527/) (2023)
- [The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2406.15885/) (2024)
- [Deep Internal Learning: Deep Learning from a Single Input](https://mlsanity.com/paper/2312.07425/) (2023)
- [A Survey of Deep Learning Audio Generation Methods](https://mlsanity.com/paper/2406.00146/) (2024)
- [Challenges in creative generative models for music: a divergence maximization perspective](https://mlsanity.com/paper/2211.08856/) (2022)
- [BlendGAN: Learning and Blending the Internal Distributions of Single Images by Spatial Image-Identity Conditioning](https://mlsanity.com/paper/2212.01589/) (2022)
- [LaughNet: synthesizing laughter utterances from waveform silhouettes and a single laughter example](https://mlsanity.com/paper/2110.04946/) (2021)
- [SpecSinGAN: Sound Effect Variation Synthesis Using Single-Image GANs](https://mlsanity.com/paper/2110.07311/) (2021)

## Abstract (arXiv)

Models for audio generation are typically trained on hours of recordings. Here, we illustrate that capturing the essence of an audio source is typically possible from as little as a few tens of seconds from a single training signal. Specifically, we present a GAN-based generative model that can be trained on one short audio signal from any domain (e.g. speech, music, etc.) and does not require pre-training or any other form of external supervision. Once trained, our model can generate random samples of arbitrary duration that maintain semantic similarity to the training waveform, yet exhibit new compositions of its audio primitives. This enables a long line of interesting applications, including generating new jazz improvisations or new a-cappella rap variants based on a single short example, producing coherent modifications to famous songs (e.g. adding a new verse to a Beatles song based solely on the original recording), filling-in of missing parts (inpainting), extending the bandwidth of a speech signal (super-resolution), and enhancing old recordings without access to any clean training example. We show that in all cases, no more than 20 seconds of training audio commonly suffice for our model to achieve state-of-the-art results. This is despite its complete lack of prior knowledge about the nature of audio signals in general.