mlsanityData to 7 Oct 2026

DiarizationLM: Speaker Diarization Post-Processing with Large Language Models

arXiv 2401.03506 ↗First posted 7 Jan 2024eess.AS cs.LG cs.SDcited by 16

Not on the map. Why?

Abstract (arXiv)

In this paper, we introduce DiarizationLM, a framework to leverage large language models (LLM) to post-process the outputs from a speaker diarization system. Various goals can be achieved with the proposed framework, such as improving the readability of the diarized transcript, or reducing the word diarization error rate (WDER). In this framework, the outputs of the automatic speech recognition (ASR) and speaker diarization systems are represented as a compact textual format, which is included in the prompt to an optionally finetuned LLM. The outputs of the LLM can be used as the refined diarization results with the desired enhancement. As a post-processing step, this framework can be easily applied to any off-the-shelf ASR and speaker diarization systems without retraining existing components. Our experiments show that a finetuned PaLM 2-S model can reduce the WDER by rel. 55.5% on the Fisher telephone conversation dataset, and rel. 44.9% on the Callhome English dataset.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 7 Jan 2024.

Builds on

42 of 73 references.

References · 42

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2017Generalized End-to-End Loss for Speaker Verificationcited by 290
  9. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  10. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  11. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  12. 2021A Review of Speaker Diarization: Recent Advances with Deep Learningcited by 122
30 more
  1. 2019End-to-End Neural Speaker Diarization with Permutation-Free Objectivescited by 86
  2. 2017Speaker Diarization with LSTMcited by 73
  3. 2020End-to-End Speaker Diarization for an Unknown Number of Speakers with Encoder-Decoder Based Attractorscited by 70
  4. 2020Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenariocited by 67
  5. 2017Deep Speaker: an End-to-End Neural Speaker Embedding Systemcited by 56
  6. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  7. 2018Fully Supervised Speaker Diarizationcited by 46
  8. 2020Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakerscited by 41
  9. 2020Auto-Tuning Spectral Clustering for Speaker Diarization Using Normalized Maximum Eigengapcited by 35
  10. 2021End-to-End Speaker-Attributed ASR with Transformercited by 35
  11. 2021Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detectioncited by 31
  12. 2019Personal VAD: Speaker-Conditioned Voice Activity Detectioncited by 24
  13. 2019Auxiliary Interference Speaker Loss for Target-Speaker Speech Recognitioncited by 16
  14. 2023Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approachcited by 15
  15. 2019Discriminative Neural Clustering for Speaker Diarisationcited by 14
  16. 2020Speaker Diarization with Lexical Informationcited by 14
  17. 2022Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddingscited by 14
  18. 2023Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correctioncited by 14
  19. 2020End-to-End Speaker Diarization as Post-Processingcited by 13
  20. 2018Multimodal Speaker Segmentation and Diarization using Lexical and Acoustic Cues via Sequence to Sequence Neural Networkscited by 12
  21. 2022Personal VAD 2.0: Optimizing Personal Voice Activity Detection for On-Device Speech Recognitioncited by 12
  22. 2020Minimum Bayes Risk Training for End-to-End Speaker-Attributed ASRcited by 11
  23. 2022EEND-SS: Joint End-to-End Neural Speaker Diarization and Speech Separation for Flexible Number of Speakerscited by 10
  24. 2022Highly Efficient Real-Time Streaming and Fully On-Device Speaker Diarization with Multi-Stage Clusteringcited by 10
  25. 2020Multi-Scale Speaker Diarization With Neural Affinity Score Fusioncited by 9
  26. 2022Augmenting Transformer-Transducer Based Speaker Change Detection With Token-Level Training Losscited by 8
  27. 2019DOVER: A Method for Combining Diarization Outputscited by 6
  28. 2023USM-SCD: Multilingual Speaker Change Detection Based on Large Pretrained Foundation Modelscited by 4
  29. 2023DiaCorrect: Error Correction Back-end For Speaker Diarizationcited by 3
  30. 2023Towards Word-Level End-to-End Neural Speaker Diarization with Auxiliary Networkcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 5; 2026: 6.

16 papers.

Sort:

By area:

  1. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognitioncited by 30
  2. 2024Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognitioncited by 10
  3. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  4. 2026D-ORCA: Dialogue-Centric Optimization for Robust Audio-Visual Captioningcited by 6
  5. 2026MOSS Transcribe Diarize Technical Reportcited by 4
  6. 2024AG-LSEC: Audio Grounded Lexical Speaker Error Correctioncited by 3
  7. 2025LLAMADRS: Evaluating Open-Source LLMs on Real Clinical Interviews--To Reason or Not to Reason?cited by 3
  8. 2024Assessing the Robustness of Spectral Clustering for Deep Speaker Diarizationcited by 2
  9. 2025Recent Trends in Distant Conversational Speech Recognition: A Review of CHiME-7 and 8 DASR Challengescited by 2
  10. 2025SEAL: Speaker Error Correction using Acoustic-conditioned Large Language Modelscited by 1
  11. 2025ASR-Synchronized Speaker-Role Diarizationcited by 0
  12. 2025Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusingcited by 0
4 more
  1. 2026Reasoning LLM Improves Speaker Recognition in Long-form TV Dramascited by 0
  2. 2026Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correctioncited by 0
  3. 2026Neural Multichannel Distant Speaker Diarization with Heavy-tailed Source Separation Modelcited by 0
  4. 2026Role-guided Speaker Deletion Verification in Clinical Psychiatry Speech Recordings with Audio Language Modelscited by 0