Audio-visual fine-tuning of audio-only ASR models
Not on the map. Why?
Abstract (arXiv)
Audio-visual automatic speech recognition (AV-ASR) models are very effective at reducing word error rates on noisy speech, but require large amounts of transcribed AV training data. Recently, audio-visual self-supervised learning (SSL) approaches have been developed to reduce this dependence on transcribed AV data, but these methods are quite complex and computationally expensive. In this work, we propose replacing these expensive AV-SSL methods with a simple and fast textitaudio-only SSL method, and then performing AV supervised fine-tuning. We show that this approach is competitive with state-of-the-art (SOTA) AV-SSL methods on the LRS3-TED benchmark task (within 0.5% absolute WER), while being dramatically simpler and more efficient (12-30x faster to pre-train). Furthermore, we show we can extend this approach to convert a SOTA audio-only ASR model into an AV model. By doing so, we match SOTA AV-SSL results, even though no AV data was used during pre-training.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 14 Dec 2023.
Builds on
29 of 38 references.
References · 29
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2018VoxCeleb2: Deep Speaker Recognitioncited by 862
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
- 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 462
17 more
- 2023Scaling Speech Technology to 1,000+ Languagescited by 401
- 2021GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10,000 Hours of Transcribed Audiocited by 391
- 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
- 2018LRS3-TED: a large-scale dataset for visual speech recognitioncited by 301
- 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 277
- 2023Google USM: Scaling Automatic Speech Recognition Beyond 100 Languagescited by 228
- 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
- 2020Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognitioncited by 185
- 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 158
- 2022Robust Self-Supervised Audio-Visual Speech Recognitioncited by 72
- 2019Recurrent Neural Network Transducer for Audio-Visual Speech Recognitioncited by 71
- 2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
- 2022VATLM: Visual-Audio-Text Pre-Training with Unified Masked Prediction for Speech Representation Learningcited by 34
- 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 31
- 2023AV-data2vec: Self-supervised Learning of Audio-Visual Speech Representations with Contextualized Target Representationscited by 24
- 2021Audio-Visual Speech Recognition is Worth 32×32×8 Voxelscited by 5
- 2015Scaling Recurrent Neural Network Language Modelscited by 2
Built on by
Citing papers per year: 2024: 2; 2025: 1.
3 papers.