# Musika! Fast Infinite Waveform Music Generation

arXiv: https://arxiv.org/abs/2208.08706 · HTML: https://mlsanity.com/paper/2208.08706/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 43 references that are in the map.
- First posted on arXiv 18 Aug 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Taming Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2012.09841/) (2020)
- [Progressive Growing of GANs for Improved Quality, Stability, and Variation](https://mlsanity.com/paper/1710.10196/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [WaveNet: A Generative Model for Raw Audio](https://mlsanity.com/paper/1609.03499/) (2016)
- [Spectral Normalization for Generative Adversarial Networks](https://mlsanity.com/paper/1802.05957/) (2018)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [HiFi-GAN: Generative Adversarial Networks for Efficient and High Fidelity Speech Synthesis](https://mlsanity.com/paper/2010.05646/) (2020)
- [Which Training Methods for GANs do actually Converge?](https://mlsanity.com/paper/1801.04406/) (2018)
- [LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speech](https://mlsanity.com/paper/1904.02882/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [MelGAN: Generative Adversarial Networks for Conditional Waveform Synthesis](https://mlsanity.com/paper/1910.06711/) (2019)
- [Geometric GAN](https://mlsanity.com/paper/1705.02894/) (2017)
- [Neural Audio Synthesis of Musical Notes with WaveNet Autoencoders](https://mlsanity.com/paper/1704.01279/) (2017)
- [Adversarial Audio Synthesis](https://mlsanity.com/paper/1802.04208/) (2018)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Projected GANs Converge Faster](https://mlsanity.com/paper/2111.01007/) (2021)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [Towards Faster and Stabilized GAN Training for High-fidelity Few-shot Image Synthesis](https://mlsanity.com/paper/2101.04775/) (2021)
- [GANSynth: Adversarial Neural Audio Synthesis](https://mlsanity.com/paper/1902.08710/) (2019)
- [SampleRNN: An Unconditional End-to-End Neural Audio Generation Model](https://mlsanity.com/paper/1612.07837/) (2016)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [Aligning Latent and Image Spaces to Connect the Unconnectable](https://mlsanity.com/paper/2104.06954/) (2021)
- [COCO-GAN: Generation by Parts via Conditional Coordinating](https://mlsanity.com/paper/1904.00284/) (2019)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [InfinityGAN: Towards Infinite-Pixel Image Synthesis](https://mlsanity.com/paper/2104.03963/) (2021)
- [GANS for Sequences of Discrete Elements with the Gumbel-softmax Distribution](https://mlsanity.com/paper/1611.04051/) (2016)

## Built on by (most cited first, 24 of 24)

- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [Mustango: Toward Controllable Text-to-Music Generation](https://mlsanity.com/paper/2311.08355/) (2023)
- [Efficient Neural Music Generation](https://mlsanity.com/paper/2305.15719/) (2023)
- [Diff-A-Riff: Musical Accompaniment Co-creation via Latent Diffusion Models](https://mlsanity.com/paper/2406.08384/) (2024)
- [Music2Latent: Consistency Autoencoders for Latent Audio Compression](https://mlsanity.com/paper/2408.06500/) (2024)
- [DITTO-2: Distilled Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2405.20289/) (2024)
- [Long-Term Rhythmic Video Soundtracker](https://mlsanity.com/paper/2305.01319/) (2023)
- [Bass Accompaniment Generation via Latent Diffusion](https://mlsanity.com/paper/2402.01412/) (2024)
- [ERNIE-Music: Text-to-Waveform Music Generation with Diffusion Models](https://mlsanity.com/paper/2302.04456/) (2023)
- [Analyzable Chain-of-Musical-Thought Prompting for High-Fidelity Music Generation](https://mlsanity.com/paper/2503.19611/) (2025)
- [Survey on the Evaluation of Generative Models in Music](https://mlsanity.com/paper/2506.05104/) (2025)
- [Detecting music deepfakes is easy but actually hard](https://mlsanity.com/paper/2405.04181/) (2024)
- [MusGO: A Community-Driven Framework For Assessing Openness in Music-Generative AI](https://mlsanity.com/paper/2507.03599/) (2025)
- [CoDiCodec: Unifying Continuous and Discrete Compressed Representations of Audio](https://mlsanity.com/paper/2509.09836/) (2025)
- [Msanii: High Fidelity Music Synthesis on a Shoestring Budget](https://mlsanity.com/paper/2301.06468/) (2023)
- [Music2Latent2: Audio Compression with Summary Embeddings and Autoregressive Decoding](https://mlsanity.com/paper/2501.17578/) (2025)
- [A Survey on Artificial Intelligence for Music Generation: Agents, Domains and Perspectives](https://mlsanity.com/paper/2210.13944/) (2022)
- [IteraTTA: An interface for exploring both text prompts and audio priors in generating music with text-to-audio models](https://mlsanity.com/paper/2307.13005/) (2023)
- [LiveBand: Live Accompaniment Generation in the Audio Domain](https://mlsanity.com/paper/2606.03803/) (2026)
- [Designing Neural Synthesizers for Low-Latency Interaction](https://mlsanity.com/paper/2503.11562/) (2025)
- [Music Boomerang: Reusing Diffusion Models for Data Augmentation and Audio Manipulation](https://mlsanity.com/paper/2507.04864/) (2025)
- [MusGU+: Toward a Musician-Centered Evaluation Framework and Discovery Tool for Generative Music AI](https://mlsanity.com/paper/2608.30940/) (2026)
- [Progressive distillation diffusion for raw music generation](https://mlsanity.com/paper/2307.10994/) (2023)

## Abstract (arXiv)

Fast and user-controllable music generation could enable novel ways of composing or performing music. However, state-of-the-art music generation systems require large amounts of data and computational resources for training, and are slow at inference. This makes them impractical for real-time interactive use. In this work, we introduce Musika, a music generation system that can be trained on hundreds of hours of music using a single consumer GPU, and that allows for much faster than real-time generation of music of arbitrary length on a consumer CPU. We achieve this by first learning a compact invertible representation of spectrogram magnitudes and phases with adversarial autoencoders, then training a Generative Adversarial Network (GAN) on this representation for a particular music domain. A latent coordinate system enables generating arbitrarily long sequences of excerpts in parallel, while a global context vector allows the music to remain stylistically coherent through time. We perform quantitative evaluations to assess the quality of the generated samples and showcase options for user control in piano and techno music generation. We release the source code and pretrained autoencoder weights at github.com/marcoppasini/musika, such that a GAN can be trained on a new music domain with a single GPU in a matter of hours.