Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition
Not on the map. Why?
Abstract (arXiv)
Recent advances in machine learning have demonstrated that multi-modal pre-training can improve automatic speech recognition (ASR) performance compared to randomly initialized models, even when models are fine-tuned on uni-modal tasks. Existing multi-modal pre-training methods for the ASR task have primarily focused on single-stage pre-training where a single unsupervised task is used for pre-training followed by fine-tuning on the downstream task. In this work, we introduce a novel method combining multi-modal and multi-task unsupervised pre-training with a translation-based supervised mid-training approach. We empirically demonstrate that such a multi-stage approach leads to relative word error rate (WER) improvements of up to 38.45% over baselines on both Librispeech and SUPERB. Additionally, we share several important findings for choosing pre-training methods and datasets.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 61 of its 79 references that are in the map. First posted on arXiv 28 Mar 2024.
Builds on
61 of 79 references.
References · 61
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Adam: A Method for Stochastic Optimizationcited by 37,240
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2016Layer Normalizationcited by 5,669
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
- 2018SlowFast Networks for Video Recognitioncited by 1,867
38 more
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
- 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
- 2021ViViT: A Video Vision Transformercited by 1,620
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
- 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,322
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 875
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
- 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 725
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 703
- 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
- 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 602
- 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 429
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
- 2018A Short Note about Kinetics-600cited by 333
- 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 308
- 2020Self-Supervised MultiModal Versatile Networkscited by 276
- 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
- 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 231
- 2019A Comparative Study on Transformer vs RNN in Speech Applicationscited by 211
- 2021UniT: Multimodal Multitask Learning with a Unified Transformercited by 195
- 2020Contrastive Learning of General-Purpose Audio Representationscited by 169
- 2020An Empirical Study on Robustness to Spurious Correlations using Pre-trained Language Modelscited by 134
- 2019End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architecturescited by 120
- 2020ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Contextcited by 97
- 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
- 2021SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Networkcited by 83
- 2022Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervisioncited by 80
- 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
- 2020Rethinking Evaluation in ASR: Are Our Models Robust Enough?cited by 73
- 2020Developing RNN-T Models Surpassing High-Performance Hybrid Models with Customization Capabilitycited by 63
- 2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
- 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 39
- 2018Toward domain-invariant speech recognition via large scale trainingcited by 35
- 2021Multimodal Self-Supervised Learning of General Audio Representationscited by 32
- 2020RNN-T Models Fail to Generalize to Out-of-Domain Audio: Causes and Solutionscited by 26
- 2023On the Connection between Pre-training Data Diversity and Fine-tuning Robustnesscited by 25
50 most cited of 61 All 61 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.