mlsanityData to 14 Oct 2026

Automatic Speech Recognition with BERT and CTC Transformers: A Review

arXiv 2410.09456 ↗First posted 12 Oct 2024cs.CL eess.AScited by 8

Not on the map. Why?

Abstract (arXiv)

This review paper provides a comprehensive analysis of recent advances in automatic speech recognition (ASR) with bidirectional encoder representations from transformers BERT and connectionist temporal classification (CTC) transformers. The paper first introduces the fundamental concepts of ASR and discusses the challenges associated with it. It then explains the architecture of BERT and CTC transformers and their potential applications in ASR. The paper reviews several studies that have used these models for speech recognition tasks and discusses the results obtained. Additionally, the paper highlights the limitations of these models and outlines potential areas for further research. All in all, this review provides valuable insights for researchers and practitioners who are interested in ASR with BERT and CTC transformers.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 28 of its 52 references that are in the map. First posted on arXiv 12 Oct 2024.

Builds on

28 of 52 references.

References · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Transformers in Vision: A Surveycited by 1,013
  3. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 76
  4. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  5. 2021Context-Aware Transformer Transducer for Speech Recognitioncited by 56
  6. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  7. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 41
  8. 2023Transformers in Speech Processing: A Surveycited by 39
  9. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
  10. 2020Non-Autoregressive Transformer ASR with CTC-Enhanced Decoder Inputcited by 24
  11. 2020Controllable Time-Delay Transformer for Real-Time Punctuation Prediction and Disfluency Detectioncited by 23
  12. 2021Innovative Bert-based Reranking Language Models for Speech Recognitioncited by 23
16 more
  1. 2024Transformers and Large Language Models for Efficient Intrusion Detection Systems: A Comprehensive Surveycited by 21
  2. 2023Deep Learning for Steganalysis of Diverse Data Types: A review of methods, taxonomy, challenges and future directionscited by 16
  3. 2021Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classificationcited by 14
  4. 2022Speech Pre-training with Acoustic Piececited by 14
  5. 2021N-Best ASR Transformer: Enhancing SLU Performance using Multiple ASR Hypothesescited by 12
  6. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  7. 2021Attention-based Multi-hypothesis Fusion for Speech Summarizationcited by 9
  8. 2021An Improved Single Step Non-autoregressive Transformer for Automatic Speech Recognitioncited by 8
  9. 2021Cross-utterance Reranking Models with BERT and Graph Convolutional Networks for Conversational Speech Recognitioncited by 7
  10. 2020An Audio-enriched BERT-based Framework for Spoken Multiple-choice Question Answeringcited by 6
  11. 2020Exploration of End-to-End ASR for OpenSTT -- Russian Open Speech-to-Text Datasetcited by 6
  12. 2021Discriminative Self-training for Punctuation Predictioncited by 6
  13. 2019Neural Zero-Inflated Quality Estimation Model For Automatic Speech Recognition Systemcited by 4
  14. 2024Artificial Intelligence for Cochlear Implants: Review of Strategies, Challenges, and Perspectivescited by 4
  15. 2020An Effective Contextual Language Modeling Framework for Speech Summarization with Augmented Featurescited by 3
  16. 2020Toward Streaming ASR with Non-Autoregressive Insertion-based Modelcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 1.

8 papers.

  1. 2025Adaptive Cyber-Attack Detection in IIoT Using Attention-Based LSTM-CNN Modelscited by 4
  2. 2025Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detectioncited by 4
  3. 2025Recent Advances in Transformer and Large Language Models for UAV Applicationscited by 3
  4. 2025Improving Pretrained YAMNet for Enhanced Speech Command Detection via Transfer Learningcited by 1
  5. 2025Enhancing Cochlear Implant Signal Coding with Scaled Dot-Product Attentioncited by 1
  6. 2025Transfer Learning-Based Deep Residual Learning for Speech Recognition in Clean and Noisy Environmentscited by 0
  7. 2025SP-MCQA: Evaluating Intelligibility of TTS Beyond the Word Levelcited by 0
  8. 2026Overcoming Decoder Inconsistencies in Whisper for Dravidian and Low-Resource Languagescited by 0