CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models
Not on the map. Why?
Abstract (arXiv)
In real-world applications, automatic speech recognition (ASR) systems must handle overlapping speech from multiple speakers and recognize rare words like technical terms. Traditional methods address multi-talker ASR and contextual biasing separately, limiting performance in complex scenarios. We propose a unified framework that combines multi-talker overlapping speech recognition and contextual biasing into a single task. Our ASR method integrates pretrained speech encoders and large language models (LLMs), using optimized finetuning strategies. We also introduce a two-stage filtering algorithm to efficiently identify relevant rare words from large biasing lists and incorporate them into the LLM's prompt input, enhancing rare word recognition. Experiments show that our approach outperforms traditional contextual biasing methods, achieving a WER of 7.9% on LibriMix and 32.9% on AMI SDM when the biasing size is 1,000, demonstrating its effectiveness in complex speech scenarios.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 25 of its 32 references that are in the map. First posted on arXiv 31 May 2025.
Builds on
25 of 32 references.
References · 25
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 156
- 2020LibriMix: An Open-Source Dataset for Generalizable Speech Separationcited by 155
- 2024WavLLM: Towards Robust and Adaptive Speech Large Language Modelcited by 109
- 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 101
- 2017Multi-talker Speech Separation with Utterance-level Permutation Invariant Training of Deep Recurrent Neural Networkscited by 98
- 2024An Embarrassingly Simple Approach for LLM with Strong ASR Capacitycited by 88
- 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
- 2020Deep Shallow Fusion for RNN-T Personalizationcited by 64
- 2024Seed-ASR: Understanding Diverse Speech and Contexts with LLM-based Speech Recognitioncited by 63
- 2023Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Networkcited by 31
13 more
- 2020Streaming end-to-end multi-talker speech recognitioncited by 29
- 2024CTC-Assisted LLM-Based Contextual ASRcited by 19
- 2022Tree-constrained Pointer Generator with Graph Neural Network Encodings for Contextual Speech Recognitioncited by 17
- 2024Large Language Model Can Transcribe Speech in Multi-Talker Scenarios with Versatile Instructionscited by 17
- 2021Multi-Speaker ASR Combining Non-Autoregressive Conformer CTC and Conditional Speaker Chaincited by 12
- 2023Two Stage Contextual Word Filtering for Context bias in Unified Streaming and Non-streaming Transducercited by 12
- 2023Adaptive Contextual Biasing for Transducer Based Streaming Speech Recognitioncited by 12
- 2024Advancing Multi-talker ASR Performance with Large Language Modelscited by 10
- 2023SURT 2.0: Advances in Transducer-based Multi-talker Speech Recognitioncited by 8
- 2024MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correctioncited by 8
- 2023ed-cec: improving rare word recognition using asr postprocessing based on error detection and context-aware error correctioncited by 7
- 2024Serialized Speech Information Guidance with Overlapped Encoding Separation for Multi-Speaker Automatic Speech Recognitioncited by 7
- 2023Graph Neural Networks for Contextual ASR with the Tree-Constrained Pointer Generatorcited by 6
Built on by
Citing papers per year: 2026: 1.
1 paper.