mlsanityData to 7 Oct 2026

SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Models

arXiv 2501.08421 ↗First posted 14 Jan 2025eess.AS cs.AI cs.CLcited by 1

Not on the map. Why?

Abstract (arXiv)

Speaker Diarization (SD) is a crucial component of modern end-to-end ASR pipelines. Traditional SD systems, which are typically audio-based and operate independently of ASR, often introduce speaker errors, particularly during speaker transitions and overlapping speech. Recently, language models including fine-tuned large language models (LLMs) have shown to be effective as a second-pass speaker error corrector by leveraging lexical context in the transcribed output. In this work, we introduce a novel acoustic conditioning approach to provide more fine-grained information from the acoustic diarizer to the LLM. We also show that a simpler constrained decoding strategy reduces LLM hallucinations, while avoiding complicated post-processing. Our approach significantly reduces the speaker error rates by 24-43% across Fisher, Callhome, and RT03-CTS datasets, compared to the first-pass Acoustic SD.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 38 references that are in the map. First posted on arXiv 14 Jan 2025.

Builds on

34 of 38 references.

References · 34

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
  2. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  3. 2023Mistral 7Bcited by 3,065
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  5. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  6. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  7. 2023Efficient Guided Generation for Large Language Modelscited by 192
  8. 2023Grammar-Constrained Decoding for Structured NLP Tasks without Finetuningcited by 186
  9. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  10. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  11. 2019Transformer-based Acoustic Modeling for Hybrid Speech Recognitioncited by 86
  12. 2019End-to-End Neural Speaker Diarization with Self-attentioncited by 74
22 more
  1. 2024SpeechVerse: A Large-scale Generalizable Audio Language Modelcited by 62
  2. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  3. 2021ECAPA-TDNN Embeddings for Speaker Diarizationcited by 47
  4. 2022Streaming Multi-Talker ASR with Token-Level Serialized Output Trainingcited by 37
  5. 2020Continuous Speech Separation with Conformercited by 36
  6. 2020Integrating end-to-end neural and clustering-based diarization: Getting the best of both worldscited by 33
  7. 2021Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detectioncited by 31
  8. 2023MossFormer: Pushing the Performance Limit of Monaural Speech Separation using Gated Single-Head Transformer with Convolution-Augmented Joint Self-Attentionscited by 27
  9. 2020Streaming Multi-speaker ASR with RNN-Tcited by 22
  10. 2024The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Modelscited by 17
  11. 2024DiarizationLM: Speaker Diarization Post-Processing with Large Language Modelscited by 16
  12. 2023Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approachcited by 15
  13. 2023Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correctioncited by 14
  14. 2018Multimodal Speaker Segmentation and Diarization using Lexical and Acoustic Cues via Sequence to Sequence Neural Networkscited by 12
  15. 2022Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clusteringcited by 10
  16. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  17. 2022A Conformer Based Acoustic Model for Robust Automatic Speech Recognitioncited by 6
  18. 2023CASA-ASR: Context-Aware Speaker-Attributed ASRcited by 6
  19. 2021Directed Speech Separation for Automatic Speech Recognition of Long Form Conversational Speechcited by 5
  20. 2023Mixture Encoder for Joint Speech Separation and Recognitioncited by 4
  21. 2024AG-LSEC: Audio Grounded Lexical Speaker Error Correctioncited by 3
  22. 2024Speakers Unembedded: Embedding-free Approach to Long-form Neural Diarizationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correctioncited by 0