mlsanityData to 7 Oct 2026

Speech Recognition by Simply Fine-tuning BERT

arXiv 2102.00291 ↗First posted 30 Jan 2021cs.SD cs.CL eess.AScited by 11

Not on the map. Why?

Abstract (arXiv)

We propose a simple method for automatic speech recognition (ASR) by fine-tuning BERT, which is a language model (LM) trained on large-scale unlabeled text data and can generate rich contextual representations. Our assumption is that given a history context sequence, a powerful LM can narrow the range of possible choices and the speech signal can be used as a simple clue. Hence, comparing to conventional ASR systems that train a powerful acoustic model (AM) from scratch, we believe that speech recognition is possible by simply fine-tuning a BERT model. As an initial study, we demonstrate the effectiveness of the proposed idea on the AISHELL dataset and show that stacking a very simple AM on top of BERT can yield reasonable performance.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 16 of its 24 references that are in the map. First posted on arXiv 30 Jan 2021.

Builds on

16 of 24 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2019Passage Re-ranking with BERTcited by 629
  7. 2015Listen, Attend and Spellcited by 542
  8. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  9. 2019K-BERT: Enabling Language Representation with Knowledge Graphcited by 303
  10. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  11. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  12. 2019Discourse-Aware Neural Extractive Text Summarizationcited by 104
4 more
  1. 2019CIF: Continuous Integrate-and-Fire for End-to-End Speech Recognitioncited by 76
  2. 2019BERT with History Answer Embedding for Conversational Question Answeringcited by 67
  3. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 38
  4. 2020TwinBERT: Distilling Knowledge to Twin-Structured BERT Models for Efficient Retrievalcited by 24

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 5; 2023: 3; 2024: 2.

11 papers.

  1. 2022Hey ASR System! Why Aren't You More Inclusive? Automatic Speech Recognition Systems' Bias and Proposed Bias Mitigation Techniques. A Literature Reviewcited by 19
  2. 2021Non-autoregressive Transformer-based End-to-end ASR using BERTcited by 13
  3. 2022A context-aware knowledge transferring strategy for CTC-based ASRcited by 12
  4. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  5. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillationcited by 11
  6. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  7. 2022Emotion Recognition with Pre-Trained Transformers Using Multimodal Signalscited by 6
  8. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  9. 2024Continuous Sign Language Recognition with Adapted Conformer via Unsupervised Pretrainingcited by 1
  10. 2023Improving Reliability of Fine-tuning with Block-wise Optimisationcited by 0
  11. 2024Could Chemical LLMs benefit from Message Passingcited by 0