# Stable Audio Open

arXiv: https://arxiv.org/abs/2407.14358 · HTML: https://mlsanity.com/paper/2407.14358/

## Key facts

- Cited by 197 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 34 references that are in the map.
- First posted on arXiv 19 Jul 2024.

## Builds on (most cited first, 27 of 27)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Scaling Rectified Flow Transformers for High-Resolution Image Synthesis](https://mlsanity.com/paper/2403.03206/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition](https://mlsanity.com/paper/1912.10211/) (2019)
- [Jukebox: A Generative Model for Music](https://mlsanity.com/paper/2005.00341/) (2020)
- [Simple and Controllable Music Generation](https://mlsanity.com/paper/2306.05284/) (2023)
- [MusicLM: Generating Music From Text](https://mlsanity.com/paper/2301.11325/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Models](https://mlsanity.com/paper/2301.12661/) (2023)
- [AudioGen: Textually Guided Audio Generation](https://mlsanity.com/paper/2209.15352/) (2022)
- [Efficient Training of Audio Transformers with Patchout](https://mlsanity.com/paper/2110.05069/) (2021)
- [Fr'echet Audio Distance: A Metric for Evaluating Music Enhancement Algorithms](https://mlsanity.com/paper/1812.08466/) (2018)
- [Fast Timing-Conditioned Latent Audio Diffusion](https://mlsanity.com/paper/2402.04825/) (2024)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [Long-form music generation with latent diffusion](https://mlsanity.com/paper/2404.10301/) (2024)
- [The Song Describer Dataset: a Corpus of Audio Captions for Music-and-Language Evaluation](https://mlsanity.com/paper/2311.10057/) (2023)
- [Controllable Music Production with Diffusion Models and Guidance Gradients](https://mlsanity.com/paper/2311.00613/) (2023)

## Built on by (most cited first, 30 of 197)

- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [UniVerse-1: Unified Audio-Video Generation via Stitching of Experts](https://mlsanity.com/paper/2509.06155/) (2025)
- [Video-Guided Foley Sound Generation with Multimodal Controls](https://mlsanity.com/paper/2411.17698/) (2024)
- [MOVA: Towards Scalable and Synchronized Video-Audio Generation](https://mlsanity.com/paper/2602.08794/) (2026)
- [Motion Anything: Any to Motion Generation](https://mlsanity.com/paper/2503.06955/) (2025)
- [Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback](https://mlsanity.com/paper/2412.15838/) (2024)
- [Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences](https://mlsanity.com/paper/2410.18881/) (2024)
- [Self-Supervised Flow Matching for Scalable Multi-Modal Synthesis](https://mlsanity.com/paper/2603.06507/) (2026)
- [SongGen: A Single Stage Auto-regressive Transformer for Text-to-Song Generation](https://mlsanity.com/paper/2502.13128/) (2025)
- [AudioX: A Unified Framework for Anything-to-Audio Generation](https://mlsanity.com/paper/2503.10522/) (2025)
- [Flow Generator Matching](https://mlsanity.com/paper/2410.19310/) (2024)
- [Both Ears Wide Open: Towards Language-Driven Spatial Audio Generation](https://mlsanity.com/paper/2410.10676/) (2024)
- [KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generation](https://mlsanity.com/paper/2502.15602/) (2025)
- [SongBloom: Coherent Song Generation via Interleaved Autoregressive Sketching and Diffusion Refinement](https://mlsanity.com/paper/2506.07634/) (2025)
- [SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers](https://mlsanity.com/paper/2411.10510/) (2024)
- [FreeAudio: Training-Free Timing Planning for Controllable Long-Form Text-to-Audio Generation](https://mlsanity.com/paper/2507.08557/) (2025)
- [FLUX that Plays Music](https://mlsanity.com/paper/2409.00587/) (2024)
- [ETTA: Elucidating the Design Space of Text-to-Audio Models](https://mlsanity.com/paper/2412.19351/) (2024)
- [DiffRhythm 2: Efficient and High Fidelity Song Generation via Block Flow Matching](https://mlsanity.com/paper/2510.22950/) (2025)
- [Fast Text-to-Audio Generation with Adversarial Post-Training](https://mlsanity.com/paper/2505.08175/) (2025)
- [Audio Super-Resolution with Latent Bridge Models](https://mlsanity.com/paper/2509.17609/) (2025)
- [Object-AVEdit: An Object-level Audio-Visual Editing Model](https://mlsanity.com/paper/2510.00050/) (2025)
- [LoVA: Long-form Video-to-Audio Generation](https://mlsanity.com/paper/2409.15157/) (2024)
- [Editing Music with Melody and Text: Using ControlNet for Diffusion Transformer](https://mlsanity.com/paper/2410.05151/) (2024)
- [Video Diffusion Transformers are In-Context Learners](https://mlsanity.com/paper/2412.10783/) (2024)
- [Guiding Audio Editing with Audio Language Model](https://mlsanity.com/paper/2509.21625/) (2025)
- [Improving Musical Accompaniment Co-creation via Diffusion Transformers](https://mlsanity.com/paper/2410.23005/) (2024)
- [Taming Text-to-Sounding Video Generation via Advanced Modality Condition and Interaction](https://mlsanity.com/paper/2510.03117/) (2025)
- [Omni2Sound: Towards Unified Video-Text-to-Audio Generation](https://mlsanity.com/paper/2601.02731/) (2026)

## Abstract (arXiv)

Open generative models are vitally important for the community, allowing for fine-tunes and serving as baselines when presenting new models. However, most current text-to-audio models are private and not accessible for artists and researchers to build upon. Here we describe the architecture and training process of a new open-weights text-to-audio model trained with Creative Commons data. Our evaluation shows that the model's performance is competitive with the state-of-the-art across various metrics. Notably, the reported FDopenl3 results (measuring the realism of the generations) showcase its potential for high-quality stereo sound synthesis at 44.1kHz.