mlsanityData to 14 Oct 2026

Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition

arXiv 2101.06699 ↗First posted 17 Jan 2021cs.CL cs.SD eess.AScited by 17

Not on the map. Why?

Abstract (arXiv)

End-to-end models have achieved impressive results on the task of automatic speech recognition (ASR). For low-resource ASR tasks, however, labeled data can hardly satisfy the demand of end-to-end models. Self-supervised acoustic pre-training has already shown its amazing ASR performance, while the transcription is still inadequate for language modeling in end-to-end models. In this work, we fuse a pre-trained acoustic encoder (wav2vec2.0) and a pre-trained linguistic encoder (BERT) into an end-to-end ASR model. The fused model only needs to learn the transfer from speech to language during fine-tuning on limited labeled data. The length of the two modalities is matched by a monotonic attention mechanism without additional parameters. Besides, a fully connected layer is introduced for the hidden mapping between modalities. We further propose a scheduled fine-tuning strategy to preserve and utilize the text context modeling ability of the pre-trained linguistic encoder. Experiments show our effective utilizing of pre-trained modules. Our model achieves better recognition performance on CALLHOME corpus (15 hours) than other end-to-end models.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Builds on 25 of its 39 references that are in the map. First posted on arXiv 17 Jan 2021.

Builds on

25 of 39 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  6. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  7. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  8. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 501
  9. 2017Non-Autoregressive Neural Machine Translationcited by 463
  10. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  11. 2016Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learningcited by 257
  12. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 88
13 more
  1. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 77
  2. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 76
  3. 2018A Comparison of Techniques for Language Model Integration in Encoder-Decoder Speech Recognitioncited by 66
  4. 2019Improving Transformer-based Speech Recognition Using Unsupervised Pre-trainingcited by 30
  5. 2020Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languagescited by 30
  6. 2020Semi-supervised ASR by End-to-end Self-trainingcited by 29
  7. 2018Multilingual End-to-End Speech Recognition with A Single Transformer on Low-Resource Languagescited by 26
  8. 2017Supervised and Unsupervised Transfer Learning for Question Answeringcited by 24
  9. 2018Adversarial Training of End-to-end Speech Recognition Using a Criticizing Language Modelcited by 12
  10. 2018A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinesecited by 10
  11. 2019Semi-supervised Sequence-to-sequence ASR using Unpaired Speech and Textcited by 10
  12. 2020A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognitioncited by 8
  13. 2020Multilingual Speech Translation with Efficient Finetuning of Pretrained Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 7; 2023: 3; 2024: 1; 2025: 0; 2026: 1.

17 papers.

Sort:

By area:

  1. 2021Learning When to Translate for Streaming Speechcited by 30
  2. 2021Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognitioncited by 16
  3. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 15
  4. 2022Improving CTC-based speech recognition via knowledge transferring from pre-trained language modelscited by 14
  5. 2022Exploring Continuous Integrate-and-Fire for Adaptive Simultaneous Speech Translationcited by 14
  6. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  7. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  8. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  9. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillationcited by 11
  10. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  11. 2024Simul-Whisper: Attention-Guided Streaming Whisper with Truncation Detectioncited by 7
  12. 2023Adapting Offline Speech Translation Models for Streaming with Future-Aware Distillation and Inferencecited by 6
5 more
  1. 2021Speech recognition for air traffic control via feature learning and end-to-end trainingcited by 2
  2. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  3. 2022A Complementary Joint Training Approach Using Unpaired Speech and Text for Low-Resource Automatic Speech Recognitioncited by 1
  4. 2021Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Modelscited by 0
  5. 2026Benchmarking and Domain Adaptation of Automatic Speech Recognition (ASR) for Adolescent Health Communication in Ghanaian Languagescited by 0