mlsanityData to 14 Oct 2026

Multi-Modal Pre-Training for Automated Speech Recognition

arXiv 2110.09890 ↗First posted 12 Oct 2021eess.AS cs.LG cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

Traditionally, research in automated speech recognition has focused on local-first encoding of audio representations to predict the spoken phonemes in an utterance. Unfortunately, approaches relying on such hyper-local information tend to be vulnerable to both local-level corruption (such as audio-frame drops, or loud noises) and global-level noise (such as environmental noise, or background noise) that has not been seen during training. In this work, we introduce a novel approach which leverages a self-supervised learning technique based on masked language modeling to compute a global, multi-modal encoding of the environment in which the utterance occurs. We then use a new deep-fusion framework to integrate this global context into a traditional ASR method, and demonstrate that the resulting method can outperform baseline methods by up to 7% on Librispeech; gains on internal datasets range from 6% (on larger models) to 45% (on smaller models).

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 21 of its 28 references that are in the map. First posted on arXiv 12 Oct 2021.

Builds on

21 of 28 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  4. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  5. 2021ViViT: A Video Vision Transformercited by 1,620
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
  8. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  9. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  10. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
  11. 2018A Short Note about Kinetics-600cited by 333
  12. 2020Self-Supervised MultiModal Versatile Networkscited by 276
9 more
  1. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  2. 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
  3. 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
  4. 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
  5. 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
  6. 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
  7. 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 32
  8. 2020RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutionscited by 26
  9. 2019Improving noise robustness of automatic speech recognition via parallel data and teacher-student learningcited by 10

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 2; 2024: 3.

10 papers.

  1. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  3. 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 37
  4. 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 27
  5. 2022Multimodal Conversational AI: A Survey of Datasets and Approachescited by 16
  6. 2022Content-Context Factorized Representations for Automated Speech Recognitioncited by 8
  7. 2023Hearing Lips in Noise: Universal Viseme-Phoneme Mapping and Transfer for Robust Audio-Visual Speech Recognitioncited by 7
  8. 2024Task Oriented Dialogue as a Catalyst for Self-Supervised Automatic Speech Recognitioncited by 1
  9. 2024Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognitioncited by 1
  10. 2024An Efficient Self-Learning Framework For Interactive Spoken Dialog Systemscited by 0