mlsanityData to 7 Oct 2026

Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognition

arXiv 2409.09785 ↗First posted 15 Sep 2024cs.CL cs.AI cs.LGcited by 10

Not on the map. Why?

Abstract (arXiv)

Given recent advances in generative AI technology, a key question is how large language models (LLMs) can enhance acoustic modeling tasks using text decoding results from a frozen, pretrained automatic speech recognition (ASR) model. To explore new capabilities in language modeling for speech processing, we introduce the generative speech transcription error correction (GenSEC) challenge. This challenge comprises three post-ASR language modeling tasks: (i) post-ASR transcription correction, (ii) speaker tagging, and (iii) emotion recognition. These tasks aim to emulate future LLM-based agents handling voice-based interfaces while remaining accessible to a broad audience by utilizing open pretrained language models or agent-based APIs. We also discuss insights from baseline evaluations, as well as lessons learned for designing future evaluations.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 34 of its 57 references that are in the map. First posted on arXiv 15 Sep 2024.

Builds on

34 of 57 references.

References · 34

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Headcited by 264
  3. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  4. 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
  5. 2020CHiME-6 Challenge:Tackling Multispeaker Speech Recognition for Unsegmented Recordingscited by 129
  6. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  7. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 71
  8. 2019A spelling correction model for end-to-end speech recognitioncited by 63
  9. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
  10. 2019Joint Speech Recognition and Speaker Diarization via Sequence Transductioncited by 49
  11. 2023N-best T5: Robust ASR Error Correction using Multiple Input Hypotheses and Constrained Decoding Spacecited by 41
  12. 2023HyPoradise: An Open Baseline for Generative Speech Recognition with Large Language Modelscited by 41
22 more
  1. 2021Turn-to-Diarize: Online Speaker Diarization Constrained by Transformer Transducer Speaker Turn Detectioncited by 31
  2. 2021Transcribe-to-Diarize: Neural Speaker Diarization for Unlimited Number of Speakers using End-to-End Speaker-Attributed ASRcited by 22
  3. 2024GenTranslate: Large Language Models are Generative Multilingual Speech and Machine Translatorscited by 22
  4. 2021Fusing ASR Outputs in Joint Training for Speech Emotion Recognitioncited by 21
  5. 2023IC3: Image Captioning by Committee Consensuscited by 20
  6. 2019DiPCo -- Dinner Party Corpuscited by 18
  7. 2023Foundation Model Assisted Automatic Speech Emotion Recognition: Transcribing, Annotating, and Augmentingcited by 17
  8. 2024DiarizationLM: Speaker Diarization Post-Processing with Large Language Modelscited by 16
  9. 2023Enhancing Speaker Diarization with Large Language Models: A Contextual Beam Search Approachcited by 15
  10. 2024Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniquescited by 15
  11. 2020Speaker Diarization with Lexical Informationcited by 14
  12. 2023Lexical Speaker Error Correction: Leveraging Language Models for Speaker Diarization Error Correctioncited by 14
  13. 2018Multimodal Speaker Segmentation and Diarization using Lexical and Acoustic Cues via Sequence to Sequence Neural Networkscited by 12
  14. 2024Crossmodal ASR Error Correction with Discrete Speech Unitscited by 9
  15. 2024MF-AED-AEC: Speech Emotion Recognition by Leveraging Multimodal Fusion, Asr Error Detection, and Asr Error Correctioncited by 8
  16. 2024LLM-based speaker diarization correction: A generalizable approachcited by 8
  17. 2024From Descriptive Richness to Bias: Unveiling the Dark Side of Generative Image Caption Enrichmentcited by 8
  18. 2023ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognitioncited by 7
  19. 2023Exploring Speaker-Related Information in Spoken Language Understanding for Better Speaker Diarizationcited by 5
  20. 2022ASR-Aware End-to-end Neural Diarizationcited by 4
  21. 2023The CHiME-7 Challenge: System Description and Performance of NeMo Team's DASR Systemcited by 3
  22. 2023Encoder-decoder multimodal speaker change detectioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 4; 2026: 2.

10 papers.

  1. 2025SoloSpeech: Enhancing Intelligibility and Quality in Target Speech Extraction through a Cascaded Generative Pipelinecited by 9
  2. 2024Revise, Reason, and Recognize: LLM-Based Emotion Recognition via Emotion-Specific Prompts and ASR Error Correctioncited by 5
  3. 2025Serialized Output Prompting for Large Language Model-based Multi-Talker Speech Recognitioncited by 5
  4. 2025Neural Spelling: A Spell-Based BCI System for Language Neural Decodingcited by 4
  5. 2024Semi-Supervised Cognitive State Classification from Speech with Multi-View Pseudo-Labelingcited by 3
  6. 2024Context and System Fusion in Post-ASR Emotion Recognition with Large Language Modelscited by 1
  7. 2024Improving Speech-based Emotion Recognition with Contextual Utterance Analysis and LLMscited by 1
  8. 2025PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decodingcited by 1
  9. 2026A Mixture-of-Experts Model for Multimodal Emotion Recognition in Conversationscited by 1
  10. 2026Sometin Beta Pass Notin: Improving Multilingual ASR for Nigerian Languages via Knowledge Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Models
  2. 2025LLM-based Generative Error Correction for Rare Words with Synthetic Data and Phonetic Context
  3. 2024Failing Forward: Improving Generative Error Correction for ASR with Synthetic Data and Retrieval Augmentation
  4. 2025GenTSE: Enhancing Target Speaker Extraction via a Coarse-to-Fine Generative Language Model
  5. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies
  6. 2025GEC-RAG: Improving Generative Error Correction via Retrieval-Augmented Generation for Automatic Speech Recognition Systems
  7. 2024Large Language Models are Efficient Learners of Noise-Robust Speech Recognition
  8. 2025Customizing Speech Recognition Model with Large Language Model Feedback