# SoundWeaver: Semantic Warm-Starting for Text-to-Audio Diffusion Serving

arXiv: https://arxiv.org/abs/2603.07865 · HTML: https://mlsanity.com/paper/2603.07865/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Audio, speech and music, one of 14 directions in Diffusion models.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- Diffusion models / [Audio, speech and music](https://mlsanity.com/diffusion/audio-speech-and-music/)

## Builds on (most cited first, 30 of 31)

- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations](https://mlsanity.com/paper/2108.01073/) (2021)
- [Consistency Models](https://mlsanity.com/paper/2303.01469/) (2023)
- [DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models](https://mlsanity.com/paper/2211.01095/) (2022)
- [Pseudo Numerical Methods for Diffusion Models on Manifolds](https://mlsanity.com/paper/2202.09778/) (2022)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [AudioLDM: Text-to-Audio Generation with Latent Diffusion Models](https://mlsanity.com/paper/2301.12503/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [DeepCache: Accelerating Diffusion Models for Free](https://mlsanity.com/paper/2312.00858/) (2023)
- [Clotho: An Audio Captioning Dataset](https://mlsanity.com/paper/1910.09387/) (2019)
- [AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretraining](https://mlsanity.com/paper/2308.05734/) (2023)
- [Timestep Embedding Tells: It's Time to Cache for Video Diffusion Model](https://mlsanity.com/paper/2411.19108/) (2024)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [Stable Audio Open](https://mlsanity.com/paper/2407.14358/) (2024)
- [Natural Language Supervision for General-Purpose Audio Representations](https://mlsanity.com/paper/2309.05767/) (2023)
- [xDiT: an Inference Engine for Diffusion Transformers (DiTs) with Massive Parallelism](https://mlsanity.com/paper/2411.01738/) (2024)
- [Retrieval-Augmented Text-to-Audio Generation](https://mlsanity.com/paper/2309.08051/) (2023)
- [ConsistencyTTA: Accelerating Diffusion-Based Text-to-Audio Generation with Consistency Distillation](https://mlsanity.com/paper/2309.10740/) (2023)
- [SmoothCache: A Universal Inference Acceleration Technique for Diffusion Transformers](https://mlsanity.com/paper/2411.10510/) (2024)
- [FlashAudio: Rectified Flows for Fast and High-Fidelity Text-to-Audio Generation](https://mlsanity.com/paper/2410.12266/) (2024)
- [FLAM: Frame-Wise Language-Audio Modeling](https://mlsanity.com/paper/2505.05335/) (2025)
- [Presto! Distilling Steps and Layers for Accelerating Music Generation](https://mlsanity.com/paper/2410.05167/) (2024)
- [Fast Text-to-Audio Generation with Adversarial Post-Training](https://mlsanity.com/paper/2505.08175/) (2025)
- [MeanAudio: Fast and Faithful Text-to-Audio Generation with Mean Flows](https://mlsanity.com/paper/2508.06098/) (2025)
- [SpeechLLM-as-Judges: Towards General and Interpretable Speech Quality Evaluation](https://mlsanity.com/paper/2510.14664/) (2025)
- [A Simple Early Exiting Framework for Accelerated Sampling in Diffusion Models](https://mlsanity.com/paper/2408.05927/) (2024)
- [IC-Cache: Efficient Large Language Model Serving via In-context Caching](https://mlsanity.com/paper/2501.12689/) (2025)
- [MoDM: Efficient Serving for Image Generation via Mixture-of-Diffusion Models](https://mlsanity.com/paper/2503.11972/) (2025)
- [SoundCTM: Unifying Score-based and Consistency Models for Full-band Text-to-Sound Generation](https://mlsanity.com/paper/2405.18503/) (2024)

## Abstract (arXiv)

Text-to-audio (T2A) diffusion models generate high-quality audio but require tens of neural function evaluations (NFEs), resulting in substantial inference latency. Existing acceleration methods primarily optimize each generation trajectory independently, even when reusable acoustic structure exists in prior examples. We introduce SoundWeaver, the first training-free framework for compositional cross example acoustic reuse. Rather than generating from pure noise, SoundWeaver constructs a prompt-conditioned acoustic prior from segments distributed across cached audios and warm-starts diffusion from an intermediate noise level. We introduce an Acoustic Composition Graph (ACG), which uses neural codec representations and segment-level identification to identify candidate cross-cache transitions and structured path search to jointly optimize prompt relevance and acoustic continuity. We further establish a warm-start error bound showing how prior quality and diffusion SNR jointly govern the admissible skip depth, motivating an adaptive contextual-bandit Skip Gater that selects the warm-start level for each request. Across T2A diffusion backbones, SoundWeaver achieves 1.48-3.74x generation speedup with only a 1K-clip cache while improving generation quality.