# WavCraft: Audio Editing and Generation with Large Language Models

arXiv: https://arxiv.org/abs/2403.09527 · HTML: https://mlsanity.com/paper/2403.09527/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 14 Mar 2024.

## Built on by (most cited first, 14 of 14)

- [Foundation Models for Music: A Survey](https://mlsanity.com/paper/2408.14340/) (2024)
- [Taming Data and Transformers for Audio Generation](https://mlsanity.com/paper/2406.19388/) (2024)
- [FlowSep: Language-Queried Sound Separation with Rectified Flow Matching](https://mlsanity.com/paper/2409.07614/) (2024)
- [Instruct-MusicGen: Unlocking Text-to-Music Editing for Music Language Models via Instruction Tuning](https://mlsanity.com/paper/2405.18386/) (2024)
- [Guiding Audio Editing with Audio Language Model](https://mlsanity.com/paper/2509.21625/) (2025)
- [AudioMorphix: Training-free audio editing with diffusion probabilistic models](https://mlsanity.com/paper/2505.16076/) (2025)
- [Bridging Paintings and Music -- Exploring Emotion based Music Generation through Paintings](https://mlsanity.com/paper/2409.07827/) (2024)
- [MMAE: A Massive Multitask Audio Editing Benchmark](https://mlsanity.com/paper/2606.07229/) (2026)
- [A Survey on Foundations and Frontiers of Multimodal Agentic Frameworks: Techniques and Applications](https://mlsanity.com/paper/2608.20379/) (2026)
- [FlowSep 2: Self-Supervised Flow Matching for Language-Queried Audio Source Separation](https://mlsanity.com/paper/2608.22111/) (2026)
- [Make Some Noise: Towards LLM audio reasoning and generation using sound tokens](https://mlsanity.com/paper/2503.22275/) (2025)
- [Emotion-Aware Speech Generation with Character-Specific Voices for Comics](https://mlsanity.com/paper/2509.15253/) (2025)
- [EmoHeal: An End-to-End System for Personalized Therapeutic Music Retrieval from Fine-grained Emotions](https://mlsanity.com/paper/2509.15986/) (2025)
- [RPPNet: Perceptually-Grouped Rhythm-Pitch Primitives for Long-Term Structure Melody Generation via Boundary-Aware Modeling](https://mlsanity.com/paper/2607.19776/) (2026)

## Abstract (arXiv)

We introduce WavCraft, a collective system that leverages large language models (LLMs) to connect diverse task-specific models for audio content creation and editing. Specifically, WavCraft describes the content of raw audio materials in natural language and prompts the LLM conditioned on audio descriptions and user requests. WavCraft leverages the in-context learning ability of the LLM to decomposes users' instructions into several tasks and tackle each task collaboratively with the particular module. Through task decomposition along with a set of task-specific models, WavCraft follows the input instruction to create or edit audio content with more details and rationales, facilitating user control. In addition, WavCraft is able to cooperate with users via dialogue interaction and even produce the audio content without explicit user commands. Experiments demonstrate that WavCraft yields a better performance than existing methods, especially when adjusting the local regions of audio clips. Moreover, WavCraft can follow complex instructions to edit and create audio content on the top of input recordings, facilitating audio producers in a broader range of applications. Our implementation and demos are available at this https://github.com/JinhuaLiang/WavCraft.