mlsanityData to 14 Oct 2026

Crossmodal ASR Error Correction with Discrete Speech Units

arXiv 2405.16677 ↗First posted 26 May 2024eess.AS cs.CL cs.SDcited by 9

Not on the map. Why?

Abstract (arXiv)

ASR remains unsatisfactory in scenarios where the speaking style diverges from that used to train ASR systems, resulting in erroneous transcripts. To address this, ASR Error Correction (AEC), a post-ASR processing approach, is required. In this work, we tackle an understudied issue: the Low-Resource Out-of-Domain (LROOD) problem, by investigating crossmodal AEC on very limited downstream data with 1-best hypothesis transcription. We explore pre-training and fine-tuning strategies and uncover an ASR domain discrepancy phenomenon, shedding light on appropriate training schemes for LROOD data. Moreover, we propose the incorporation of discrete speech units to align with and enhance the word embeddings for improving AEC quality. Results from multiple corpora and several evaluation metrics demonstrate the feasibility and efficacy of our proposed AEC approach on LROOD data as well as its generalizability and superiority on large-scale data. Finally, a study on speech emotion recognition confirms that our model produces ASR error-robust transcripts suitable for downstream applications.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Builds on 25 of its 38 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

25 of 38 references.

References · 25

  1. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  5. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,475
  6. 2019Common Voice: A Massively-Multilingual Speech Corpuscited by 1,384
  7. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  8. 2022Comparative layer-wise analysis of self-supervised speech modelscited by 158
  9. 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
  10. 2023Generative Speech Recognition Error Correction with Large Language Models and Task-Activating Promptingcited by 72
  11. 2019A spelling correction model for end-to-end speech recognitioncited by 63
  12. 2023Whispering LLaMA: A Cross-Modal Generative Error Correction Framework for Speech Recognitioncited by 56
13 more
  1. 2020ASR Error Correction and Domain Adaptation Using Machine Translationcited by 48
  2. 2022Exploration of A Self-Supervised Speech Model: A Study on Emotional Corporacited by 31
  3. 2020Improving Readability for Automatic Speech Recognition Transcriptioncited by 22
  4. 2024It's Never Too Late: Fusing Acoustic Information into Large Language Models for Automatic Speech Recognitioncited by 22
  5. 2022Analyzing Acoustic Word Embeddings from Pre-trained Self-supervised Speech Modelscited by 21
  6. 2023Self-supervised representations in speech-based depression detectioncited by 18
  7. 2024Layer-Wise Analysis of Self-Supervised Acoustic Word Embeddings: A Study on Speech Emotion Recognitioncited by 8
  8. 2020Analyzing autoencoder-based acoustic word embeddingscited by 7
  9. 2023Exploring Data Augmentation for Code Generation Taskscited by 7
  10. 2023ASR and Emotional Speech: A Word-Level Investigation of the Mutual Impact of Speech and Emotion Recognitioncited by 7
  11. 2022Cross-Modal ASR Post-Processing System for Error Correction and Utterance Rejectioncited by 6
  12. 2021Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognitioncited by 4
  13. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2

Built on by

Cited by year

Citing papers per year: 2024: 5; 2025: 4.

9 papers.

  1. 2024Speech Emotion Recognition with ASR Transcripts: A Comprehensive Study on Word Error Rate and Fusion Techniquescited by 15
  2. 2024Large Language Model Based Generative Error Correction: A Challenge and Baselines for Speech Recognition, Speaker Tagging, and Emotion Recognitioncited by 11
  3. 2025Better Pseudo-labeling with Multi-ASR Fusion and Error Correction by SpeechLLMcited by 6
  4. 2025Segmentation-Variant Codebooks for Preservation of Paralinguistic and Prosodic Informationcited by 4
  5. 2024Robust ASR Error Correction with Conservative Data Filteringcited by 3
  6. 2024Context and System Fusion in Post-ASR Emotion Recognition with Large Language Modelscited by 1
  7. 2024Enhancing AAC Software for Dysarthric Speakers in e-Health Settings: An Evaluation Using TORGOcited by 1
  8. 2025PMF-CEC: Phoneme-augmented Multimodal Fusion for Context-aware ASR Error Correction with Error-specific Selective Decodingcited by 1
  9. 2025PARCO: Phoneme-Augmented Robust Contextual ASR via Contrastive Entity Disambiguationcited by 0