# FireRedAudio: A General-Purpose Audio Language Model with Decoupled Continuous Representations for Understanding and Generation

arXiv: https://arxiv.org/abs/2608.24168 · HTML: https://mlsanity.com/paper/2608.24168/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 54 references that are in the map.
- First posted on arXiv 25 Aug 2026.

## Builds on (most cited first, 30 of 50)

- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Common Voice: A Massively-Multilingual Speech Corpus](https://mlsanity.com/paper/1912.06670/) (2019)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Unsupervised Cross-lingual Representation Learning for Speech Recognition](https://mlsanity.com/paper/2006.13979/) (2020)
- [Qwen3-Omni Technical Report](https://mlsanity.com/paper/2509.17765/) (2025)
- [FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech](https://mlsanity.com/paper/2205.12446/) (2022)
- [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017)
- [F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching](https://mlsanity.com/paper/2410.06885/) (2024)
- [Seed-TTS: A Family of High-Quality Versatile Speech Generation Models](https://mlsanity.com/paper/2406.02430/) (2024)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models](https://mlsanity.com/paper/2412.10117/) (2024)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [WenetSpeech: A 10000+ Hours Multi-domain Mandarin Corpus for Speech Recognition](https://mlsanity.com/paper/2110.03370/) (2021)
- [Qwen3.5-Omni Technical Report](https://mlsanity.com/paper/2604.15804/) (2026)
- [Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis](https://mlsanity.com/paper/2306.00814/) (2023)
- [CosyVoice 3: Towards In-the-wild Speech Generation via Scaling-up and Post-training](https://mlsanity.com/paper/2505.17589/) (2025)
- [Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition](https://mlsanity.com/paper/2206.08317/) (2022)
- [AISHELL-2: Transforming Mandarin ASR Research Into Industrial Scale](https://mlsanity.com/paper/1808.10583/) (2018)
- [Qwen3-TTS Technical Report](https://mlsanity.com/paper/2601.15621/) (2026)
- [Step-Audio 2 Technical Report](https://mlsanity.com/paper/2507.16632/) (2025)
- [MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.04779/) (2025)
- [Codec Does Matter: Exploring the Semantic Shortcoming of Codec for Audio Language Model](https://mlsanity.com/paper/2408.17175/) (2024)
- [FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications](https://mlsanity.com/paper/2409.03283/) (2024)
- [MiMo-Audio: Audio Language Models are Few-Shot Learners](https://mlsanity.com/paper/2512.23808/) (2025)

## Built on by (most cited first, 2 of 2)

- [Qwen-Audio-3.0-ASR Technical Report](https://mlsanity.com/paper/2609.07549/) (2026)
- [Training a Compact, Strong, and Realistic Text-to-Speech Model](https://mlsanity.com/paper/2610.14776/) (2026)

## Abstract (arXiv)

A unified audio model must recognize and understand linguistic, paralinguistic, and environmental information while supporting speech synthesis and editing. A key challenge is representation: understanding favors compact features suited to long-context modeling, whereas speech generation requires reconstructible features that preserve fine-grained acoustic detail. We introduce FireRedAudio, a general-purpose audio language model with a shared 9B-parameter LLM. To the best of our knowledge, it is the first publicly disclosed unified audio-language model to provide separate continuous input representations for understanding and generation within a single trainable autoregressive LLM. Audio to be recognized or analyzed is processed by a dedicated Audio Encoder, while speech inputs for generation use a RedAE-based pathway. The LLM directly generates text or conditions a flow-matching DiT to produce continuous acoustic latents. Through progressive multitask training, FireRedAudio supports ASR and audio understanding, with the latter extending to recordings of up to one hour, as well as zero-shot TTS, Instruct TTS, and semantic and acoustic speech editing. Its structured organization of long-form audio achieves second-level timestamp accuracy. Across comprehensive evaluations, FireRedAudio achieves competitive or leading performance in audio understanding and multilingual ASR, strong content accuracy and speaker preservation in zero-shot TTS, leading instruction following in Instruct TTS, and substantial improvements over Ming-UniAudio-Edit in both semantic and acoustic speech editing. These results demonstrate the viability of decoupled continuous input representations for unifying audio understanding and continuous-latent speech generation in a model of moderate scale. Our code is available at https://github.com/FireRedTeam/FireRedAudio.