Masked Audio Text Encoders are Effective Multi-Modal Rescorers
Not on the map. Why?
Abstract (arXiv)
Masked Language Models (MLMs) have proven to be effective for second-pass rescoring in Automatic Speech Recognition (ASR) systems. In this work, we propose Masked Audio Text Encoder (MATE), a multi-modal masked language model rescorer which incorporates acoustic representations into the input space of MLM. We adopt contrastive learning for effectively aligning the modalities by learning shared representations. We show that using a multi-modal rescorer is beneficial for domain generalization of the ASR system when target domain data is unavailable. MATE reduces word error rate (WER) by 4%-16% on in-domain, and 3%-7% on out-of-domain datasets, over the text-only baseline. Additionally, with very limited amount of training data (0.8 hours), MATE achieves a WER reduction of 8%-23% over the first-pass baseline.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 11 May 2023.
Builds on
39 of 58 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2015Listen, Attend and Spellcited by 542
27 more
- 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 455
- 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
- 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
- 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
- 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
- 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
- 2019BERT has a Mouth, and It Must Speak: BERT as a Markov Random Field Language Modelcited by 184
- 2015End-to-End Attention-based Large Vocabulary Speech Recognitioncited by 148
- 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
- 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
- 2015EESEN: End-to-End Speech Recognition using Deep RNN Models and WFST-based Decodingcited by 73
- 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
- 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
- 2014End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Resultscited by 68
- 2019Two-Pass End-to-End Speech Recognitioncited by 62
- 2020Deliberation Model Based Two-Pass End-to-End Speech Recognitioncited by 57
- 2021FastCorrect: Fast Error Correction with Edit Alignment for Automatic Speech Recognitioncited by 43
- 2020Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignmentcited by 37
- 2021Societal Biases in Retrieved Contents: Measurement Framework and Adversarial Mitigation for BERT Rankerscited by 34
- 2022RescoreBERT: Discriminative Speech Recognition Rescoring with BERTcited by 34
- 2021Transformer Based Deliberation for Two-Pass Speech Recognitioncited by 29
- 2022WAVPROMPT: Towards Few-Shot Spoken Language Understanding with Frozen Language Modelscited by 25
- 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
- 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
- 2019Audio-attention discriminative language model for ASR rescoringcited by 13
- 2022Improving Deliberation by Text-Only and Semi-Supervised Trainingcited by 10
- 2023KG-ECO: Knowledge Graph Enhanced Entity Correction for Query Rewritingcited by 3
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.