# Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognition

arXiv: https://arxiv.org/abs/2407.04675 · HTML: https://mlsanity.com/paper/2407.04675/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Jul 2024.

## Built on by (most cited first, 30 of 63)

- [Qwen2.5-Omni Technical Report](https://mlsanity.com/paper/2503.20215/) (2025)
- [Step-Audio 2 Technical Report](https://mlsanity.com/paper/2507.16632/) (2025)
- [MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantization](https://mlsanity.com/paper/2501.01108/) (2025)
- [Language Model Can Listen While Speaking](https://mlsanity.com/paper/2408.02622/) (2024)
- [Audio-CoT: Exploring Chain-of-Thought Reasoning in Large Audio Language Model](https://mlsanity.com/paper/2501.07246/) (2025)
- [Recent Advances in Discrete Speech Tokens: A Review](https://mlsanity.com/paper/2502.06490/) (2025)
- [SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation](https://mlsanity.com/paper/2505.17060/) (2025)
- [Fun-ASR Technical Report](https://mlsanity.com/paper/2509.12508/) (2025)
- [SpeechR: A Benchmark for Speech Reasoning in Large Audio-Language Models](https://mlsanity.com/paper/2508.02018/) (2025)
- [TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch](https://mlsanity.com/paper/2412.08237/) (2024)
- [Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM](https://mlsanity.com/paper/2409.15905/) (2024)
- [Seed LiveInterpret 2.0: End-to-end Simultaneous Speech-to-speech Translation with Your Voice](https://mlsanity.com/paper/2507.17527/) (2025)
- [Low-Resource Domain Adaptation for Speech LLMs via Text-Only Fine-Tuning](https://mlsanity.com/paper/2506.05671/) (2025)
- [Speech LLMs in Low-Resource Scenarios: Data Volume Requirements and the Impact of Pretraining on High-Resource Languages](https://mlsanity.com/paper/2508.05149/) (2025)
- [Covo-Audio Technical Report](https://mlsanity.com/paper/2602.09823/) (2026)
- [Cross-Modal Knowledge Distillation for Speech Large Language Models](https://mlsanity.com/paper/2509.14930/) (2025)
- [MNV-17: A High-Quality Performative Mandarin Dataset for Nonverbal Vocalization Recognition in Speech](https://mlsanity.com/paper/2509.18196/) (2025)
- [Uni-ASR: Unified LLM-Based Architecture for Non-Streaming and Streaming Automatic Speech Recognition](https://mlsanity.com/paper/2603.11123/) (2026)
- [SecoustiCodec: Cross-Modal Aligned Streaming Single-Codecbook Speech Codec](https://mlsanity.com/paper/2508.02849/) (2025)
- [Mega-ASR: Towards In-the-wild^2 Speech Recognition via Scaling up Real-world Acoustic Simulation](https://mlsanity.com/paper/2605.19833/) (2026)
- [MERaLiON-SpeechEncoder: Towards a Speech Foundation Model for Singapore and Beyond](https://mlsanity.com/paper/2412.11538/) (2024)
- [Audio Interaction Model](https://mlsanity.com/paper/2606.05121/) (2026)
- [NEST-RQ: Next Token Prediction for Speech Self-Supervised Pre-Training](https://mlsanity.com/paper/2409.08680/) (2024)
- [TouchASP: Elastic Automatic Speech Perception that Everyone Can Touch](https://mlsanity.com/paper/2412.15622/) (2024)
- [OrchMLLM: Orchestrate Multimodal Data with Batch Post-Balancing to Accelerate Multimodal Large Language Model Training](https://mlsanity.com/paper/2503.23830/) (2025)
- [Large Language models for Time Series Analysis: Techniques, Applications, and Challenges](https://mlsanity.com/paper/2506.11040/) (2025)
- [Explore the Reinforcement Learning for the LLM based ASR and TTS system](https://mlsanity.com/paper/2509.18569/) (2025)
- [Task Vector in TTS: Toward Emotionally Expressive Dialectal Speech Synthesis](https://mlsanity.com/paper/2512.18699/) (2025)
- [Tiny-Align: Bridging Automatic Speech Recognition and Large Language Model on the Edge](https://mlsanity.com/paper/2411.13766/) (2024)
- [Temporal-Aware Iterative Speech Model for Dementia Detection](https://mlsanity.com/paper/2510.00030/) (2025)

## Abstract (arXiv)

Modern automatic speech recognition (ASR) model is required to accurately transcribe diverse speech signals (from different domains, languages, accents, etc) given the specific contextual information in various application scenarios. Classic end-to-end models fused with extra language models perform well, but mainly in data matching scenarios and are gradually approaching a bottleneck. In this work, we introduce Seed-ASR, a large language model (LLM) based speech recognition model. Seed-ASR is developed based on the framework of audio conditioned LLM (AcLLM), leveraging the capabilities of LLMs by inputting continuous speech representations together with contextual information into the LLM. Through stage-wise large-scale training and the elicitation of context-aware capabilities in LLM, Seed-ASR demonstrates significant improvement over end-to-end models on comprehensive evaluation sets, including multiple domains, accents/dialects and languages. Additionally, Seed-ASR can be further deployed to support specific needs in various scenarios without requiring extra language models. Compared to recently released large ASR models, Seed-ASR achieves 10%-40% reduction in word (or character, for Chinese) error rates on Chinese and English public test sets, further demonstrating its powerful performance.