# Live Music Models

arXiv: https://arxiv.org/abs/2508.04651 · HTML: https://mlsanity.com/paper/2508.04651/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 63 references that are in the map.
- First posted on arXiv 6 Aug 2025.

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [CoCa: Contrastive Captioners are Image-Text Foundation Models](https://mlsanity.com/paper/2205.01917/) (2022)
- [Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation](https://mlsanity.com/paper/2108.12409/) (2021)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [AudioLM: a Language Modeling Approach to Audio Generation](https://mlsanity.com/paper/2209.03143/) (2022)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Scaling Up Models and Data with textttt5x and textttseqio](https://mlsanity.com/paper/2203.17189/) (2022)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [DDSP: Differentiable Digital Signal Processing](https://mlsanity.com/paper/2001.04643/) (2020)
- [UniAudio: An Audio Foundation Model Toward Universal Audio Generation](https://mlsanity.com/paper/2310.00704/) (2023)
- [MusicLDM: Enhancing Novelty in Text-to-Music Generation Using Beat-Synchronous Mixup Strategies](https://mlsanity.com/paper/2308.01546/) (2023)
- [Stay on topic with Classifier-Free Guidance](https://mlsanity.com/paper/2306.17806/) (2023)
- [Long-form music generation with latent diffusion](https://mlsanity.com/paper/2404.10301/) (2024)
- [MuLan: A Joint Embedding of Music Audio and Natural Language](https://mlsanity.com/paper/2208.12415/) (2022)
- [RAVE: A variational autoencoder for fast and high-quality neural audio synthesis](https://mlsanity.com/paper/2111.05011/) (2021)
- [DITTO: Diffusion Inference-Time T-Optimization for Music Generation](https://mlsanity.com/paper/2401.12179/) (2024)
- [YuE: Scaling Open Foundation Models for Long-Form Music Generation](https://mlsanity.com/paper/2503.08638/) (2025)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [Codified audio language modeling learns useful representations for music information retrieval](https://mlsanity.com/paper/2107.05677/) (2021)
- [The challenge of realistic music generation: modelling raw audio at scale](https://mlsanity.com/paper/1806.10474/) (2018)
- [ACE-Step: A Step Towards Music Generation Foundation Model](https://mlsanity.com/paper/2506.00045/) (2025)

## Built on by (most cited first, 10 of 10)

- [CMI-RewardBench: Evaluating Music Reward Models with Compositional Multimodal Instruction](https://mlsanity.com/paper/2603.00610/) (2026)
- [Towards Real-Time Human-AI Musical Co-Performance: Accompaniment Generation with Latent Diffusion Models and MAX/MSP](https://mlsanity.com/paper/2604.07612/) (2026)
- [LiveBand: Live Accompaniment Generation in the Audio Domain](https://mlsanity.com/paper/2606.03803/) (2026)
- [Live Music Diffusion Models: Efficient Fine-Tuning and Post-Training of Interactive Diffusion Music Generators](https://mlsanity.com/paper/2605.22717/) (2026)
- [What's a Credit Worth? A Market Framework for Attribution-Aware Compensation in Generative Music](https://mlsanity.com/paper/2607.00641/) (2026)
- [The Ghost in the Keys: A Disklavier Demo for Human-AI Musical Co-Creativity](https://mlsanity.com/paper/2511.01663/) (2025)
- [Auditing Training Data in Generative Music Models via Black-Box Membership Inference](https://mlsanity.com/paper/2605.29202/) (2026)
- [Noisy-Channel Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2607.05198/) (2026)
- [A Quantized Native Runtime for On-Device Semantic Audio Generation](https://mlsanity.com/paper/2607.08526/) (2026)
- [TrueMuse: A Benchmark for Data Attribution in Text-to-Music Models](https://mlsanity.com/paper/2610.00835/) (2026)

## Abstract (arXiv)

We introduce a new class of generative models for music called live music models that produce a continuous stream of music in real-time with synchronized user control. We release Magenta RealTime, an open-weights live music model that can be steered using text or audio prompts to control acoustic style. On automatic metrics of music quality, Magenta RealTime outperforms other open-weights music generation models, despite using fewer parameters and offering first-of-its-kind live generation capabilities. We also release Lyria RealTime, an API-based model with extended controls, offering access to our most powerful model with wide prompt coverage. These models demonstrate a new paradigm for AI-assisted music creation that emphasizes human-in-the-loop interaction for live music performance.