mlsanityData to 7 Oct 2026

CALM: Joint Contextual Acoustic-Linguistic Modeling for Personalization of Multi-Speaker ASR

arXiv 2601.22792 ↗First posted 30 Jan 2026eess.AS cs.CL cs.SDcited by 0

Not on the map. Why?

Abstract (arXiv)

We present CALM, a joint Contextual Acoustic-Linguistic Modeling framework for multi-speaker automatic speech recognition (ASR). In personalized AI scenarios, the joint availability of acoustic and linguistic cues naturally motivates the integration of target-speaker conditioning with contextual biasing in overlapping conversations. CALM implements this integration in an end-to-end framework through speaker embedding-driven target-speaker extraction and dynamic vocabulary-based contextual biasing. We evaluate CALM on simulated English (LibriSpeechMix) and Japanese (Corpus of Spontaneous Japanese mixtures, CSJMix). On two-speaker mixtures, CALM reduces biased word error rate (B-WER) from 12.7 to 4.7 on LibriSpeech2Mix and biased character error rate (B-CER) from 16.6 to 8.4 on CSJMix2 (eval3), demonstrating the effectiveness of joint acoustic-linguistic modeling across languages. We additionally report results on the AMI corpus (IHM-mix condition) to validate performance on standardized speech mixtures.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 38 of its 47 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

38 of 47 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  3. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  6. 2020ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verificationcited by 606
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  8. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  10. 2021Contextualized Streaming End-to-End Speech Recognition with Trie-Based Deep Biasing and Shallow Fusioncited by 100
  11. 2020Serialized Output Training for End-to-End Overlapped Speech Recognitioncited by 87
  12. 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 67
26 more
  1. 2021M2MeT: The ICASSP 2022 Multi-Channel Multi-Party Meeting Transcription Challengecited by 67
  2. 2022Contextual Adapters for Personalized Speech Recognition in Neural Transducerscited by 57
  3. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  4. 2023Contextualized End-to-End Speech Recognition with Contextual Phrase Prediction Networkcited by 31
  5. 2025OWSM v4: Improving Open Whisper-Style Speech Models via Data Scaling and Cleaningcited by 27
  6. 2024Contextualized Automatic Speech Recognition with Attention-Based Bias Phrase Boosted Beam Searchcited by 19
  7. 2022Adapting self-supervised models to multi-talker speech recognition using speaker embeddingscited by 18
  8. 2023Robust Acoustic and Semantic Contextual Biasing in Neural Transducers for Speech Recognitioncited by 18
  9. 2024NOTSOFAR-1 Challenge: New Datasets, Baseline, and Tasks for Distant Meeting Transcriptioncited by 18
  10. 2024CTC-Assisted LLM-Based Contextual ASRcited by 18
  11. 2024Improving ASR Contextual Biasing with Guided Attentioncited by 17
  12. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  13. 2023CopyNE: Better Contextual ASR by Copying Named Entitiescited by 13
  14. 2024Empowering Whisper as a Joint Multi-Talker and Target-Talker Speech Recognition Systemcited by 11
  15. 2023Conformer-based Target-Speaker Automatic Speech Recognition for Single-Channel Audiocited by 10
  16. 2024Contextualized End-to-end Automatic Speech Recognition with Intermediate Biasing Losscited by 9
  17. 2024Target Speaker ASR with Whispercited by 9
  18. 2022Streaming Target-Speaker ASR with Neural Transducercited by 8
  19. 2024Contextualized Automatic Speech Recognition with Dynamic Vocabularycited by 8
  20. 2023Meeting Recognition with Continuous Speech Separation and Transcription-Supported Diarizationcited by 5
  21. 2024NTT Multi-Speaker ASR System for the DASR Task of CHiME-8 Challengecited by 5
  22. 2023End-to-End Joint Target and Non-Target Speakers ASRcited by 4
  23. 2024SQ-Whisper: Speaker-Querying based Whisper Model for Target-Speaker ASRcited by 4
  24. 2025Unifying Diarization, Separation, and ASR with Multi-Speaker Encodercited by 4
  25. 2025DYNAC: Dynamic Vocabulary based Non-Autoregressive Contextualization for Speech Recognitioncited by 1
  26. 2025CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Modelscited by 1

Built on by

Not cited yet.