# AudioChat: Unified Audio Storytelling, Editing, and Understanding with Transfusion Forcing

arXiv: https://arxiv.org/abs/2602.17097 · HTML: https://mlsanity.com/paper/2602.17097/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2026.

## Built on by (most cited first, 4 of 4)

- [MMAE: A Massive Multitask Audio Editing Benchmark](https://mlsanity.com/paper/2606.07229/) (2026)
- [Bagpiper-TTS: Natural Language Guided Universal Speech Synthesis](https://mlsanity.com/paper/2606.22811/) (2026)
- [CounterFlow: A Two-Phase Inference-Time Sampling for Counterfactual Video Foley Generation](https://mlsanity.com/paper/2605.18916/) (2026)
- [YODAS v3: Over 1 Million Hours of High-Bandwidth, Stereophonic, Multilingual Speech](https://mlsanity.com/paper/2609.29448/) (2026)

## Abstract (arXiv)

Despite recent breakthroughs, audio foundation models struggle in processing complex multi-source acoustic scenes. We refer to this challenging domain as audio stories, which can have multiple speakers and background/foreground sound effects. Compared to traditional audio processing tasks, audio stories introduce new layers of semantic, temporal, and physical complexity. To address this challenge, we propose AudioChat, a framework for developing audio foundation models that can generate, edit, and understand audio stories. AudioChat introduces a new paradigm in which LLM-based toolcalling agents simulate interactions between users and the system, and these simulated dialogues are used as training data. We also introduce a novel Audio Transfusion Forcing objective to train the AudioChat model, allowing it to simultaneously decompose high-level instructions via structured chain-of-thought reasoning and perform interactive multi-turn audio understanding/generation. To evaluate generation and editing performance, we develop three new metrics that directly measure task performance instead of relying upon distribution-based scoring. We highly encourage readers to visit our demo to better understand the capabilities of AudioChat: https://wanchichen.github.io/audiochat/.