# MOSS-Audio Technical Report

arXiv: https://arxiv.org/abs/2606.01802 · HTML: https://mlsanity.com/paper/2606.01802/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 60 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 30 of 50)

- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021)
- [Bag of Tricks for Efficient Text Classification](https://mlsanity.com/paper/1607.01759/) (2016)
- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Large-scale Contrastive Language-Audio Pretraining with Feature Fusion and Keyword-to-Caption Augmentation](https://mlsanity.com/paper/2211.06687/) (2022)
- [High Fidelity Neural Audio Compression](https://mlsanity.com/paper/2210.13438/) (2022)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Models](https://mlsanity.com/paper/2311.07919/) (2023)
- [Moshi: a speech-text foundation model for real-time dialogue](https://mlsanity.com/paper/2410.00037/) (2024)
- [Qwen2-Audio Technical Report](https://mlsanity.com/paper/2407.10759/) (2024)
- [SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities](https://mlsanity.com/paper/2305.11000/) (2023)
- [SALMONN: Towards Generic Hearing Abilities for Large Language Models](https://mlsanity.com/paper/2310.13289/) (2023)
- [High-Fidelity Audio Compression with Improved RVQGAN](https://mlsanity.com/paper/2306.06546/) (2023)
- [Scaling Speech Technology to 1,000+ Languages](https://mlsanity.com/paper/2305.13516/) (2023)
- [BEATs: Audio Pre-Training with Acoustic Tokenizers](https://mlsanity.com/paper/2212.09058/) (2022)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2410.19168/) (2024)
- [WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research](https://mlsanity.com/paper/2303.17395/) (2023)
- [GLM-4-Voice: Towards Intelligent and Human-Like End-to-End Spoken Chatbot](https://mlsanity.com/paper/2412.02612/) (2024)
- [Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming](https://mlsanity.com/paper/2408.16725/) (2024)
- [Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities](https://mlsanity.com/paper/2402.01831/) (2024)
- [SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing](https://mlsanity.com/paper/2110.07205/) (2021)
- [GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilities](https://mlsanity.com/paper/2406.11768/) (2024)
- [Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilities](https://mlsanity.com/paper/2503.03983/) (2025)
- [Spirit LM: Interleaved Spoken and Written Language Model](https://mlsanity.com/paper/2402.05755/) (2024)
- [Qwen3-ASR Technical Report](https://mlsanity.com/paper/2601.21337/) (2026)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [MMSU: A Massive Multi-task Spoken Language Understanding and Reasoning Benchmark](https://mlsanity.com/paper/2506.04779/) (2025)

## Built on by (most cited first, 11 of 11)

- [SPEARBench: A Benchmark for Naturalness Evaluation in Streaming Speech-to-Speech Language Models](https://mlsanity.com/paper/2607.05365/) (2026)
- [Stable Autoregressive Speech Generation with Low-Frame-Rate High-Dimensional Continuous Tokens](https://mlsanity.com/paper/2607.29363/) (2026)
- [MuseCritic: Learning Multi-Aspect Song Rewards through Natural-Language Aesthetic Critiques](https://mlsanity.com/paper/2608.11755/) (2026)
- [Listen, Reason, and Segment: Aligning LALMs with Editorial Judgment for Media Chapterization](https://mlsanity.com/paper/2608.16539/) (2026)
- [Augmenting Large Audio-Language Models with Frame-Level Grounding for Fine-Grained Temporal Perception](https://mlsanity.com/paper/2609.15215/) (2026)
- [Audio LLMs Know When They Can't Hear You](https://mlsanity.com/paper/2609.30625/) (2026)
- [On Temporal Binding in Large Audio Language Models](https://mlsanity.com/paper/2609.34806/) (2026)
- [What Comes Next? Omni-StoryBench for Evaluating Story-Grounded Omnimodal Generation](https://mlsanity.com/paper/2609.37317/) (2026)
- [SE-ADD: Self-Evolving Audio Deepfake Detection with Mistake-Driven Supervision](https://mlsanity.com/paper/2609.39679/) (2026)
- [SEAR: Spoofing Evidence-Grounded Audio Reasoning Benchmark for Audio Language Models](https://mlsanity.com/paper/2609.39847/) (2026)
- [ScreenASR-Bench: A Tiered Benchmark for Speech Recognition with Visual Context](https://mlsanity.com/paper/2610.16260/) (2026)

## Abstract (arXiv)

MOSS-Audio is a unified audio-language model for speech, environmental sound, and music understanding, supporting audio captioning, time-aware question answering, timestamped transcription, and audio-grounded reasoning. MOSS-Audio couples a dedicated audio encoder with a modality adapter and a large language model: the encoder produces 12.5 Hz temporal representations, the adapter projects them into the decoder space, and the decoder generates autoregressive text outputs. Two design choices are central to the system: DeepStack cross-layer feature injection, which exposes the decoder to acoustic information from multiple encoder depths, and time markers, which provide explicit temporal cues by inserting timestamp markers into the audio-token stream. At the data level, we design an event-preserving audio annotation pipeline that segments raw audio at coherent event boundaries, applies branch-specific annotation to speech, music, and general audio, and merges the results into unified captions for pretraining. The intermediate branch-specific captions are further retained to support the construction of task-oriented SFT data. The model is pretrained on large-scale audio-language data, with time-aware objectives incorporated to support temporal grounding, and then undergoes multi-stage post-training to enhance instruction following and audio-grounded reasoning. We release 4B and 8B variants in both Instruct and Thinking configurations. MOSS-Audio achieves strong performance across general audio understanding, speech captioning, ASR, and timestamped ASR, positioning it as a promising understanding foundation for future voice agents.