mlsanityData to 14 Oct 2026

BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Model

arXiv 2210.16663 ↗First posted 29 Oct 2022eess.AS cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

This paper presents BERT-CTC, a novel formulation of end-to-end speech recognition that adapts BERT for connectionist temporal classification (CTC). Our formulation relaxes the conditional independence assumptions used in conventional CTC and incorporates linguistic knowledge through the explicit output dependency obtained by BERT contextual embedding. BERT-CTC attends to the full contexts of the input and hypothesized output sequences via the self-attention mechanism. This mechanism encourages a model to learn inner/inter-dependencies between the audio and token representations while maintaining CTC's training efficiency. During inference, BERT-CTC combines a mask-predict algorithm with CTC decoding, which iteratively refines an output sequence. The experimental results reveal that BERT-CTC improves over conventional approaches across variations in speaking styles and languages. Finally, we show that the semantic representations in BERT-CTC are beneficial towards downstream spoken language understanding tasks.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 62 of its 89 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

62 of 89 references.

References · 62

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  8. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  10. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  11. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  12. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
38 more
  1. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  2. 2018Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidatescited by 639
  3. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 558
  4. 2015Listen, Attend and Spellcited by 543
  5. 2017Non-Autoregressive Neural Machine Translationcited by 463
  6. 2015Attention-Based Models for Speech Recognitioncited by 401
  7. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 373
  8. 2014Deep Speech: Scaling up end-to-end speech recognitioncited by 354
  9. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 263
  10. 2019Levenshtein Transformercited by 199
  11. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 198
  12. 2017State-of-the-art Speech Recognition With Sequence-to-Sequence Modelscited by 196
  13. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 150
  14. 2020Recent Developments on ESPnet Toolkit Boosted by Conformercited by 132
  15. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 111
  16. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  17. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  18. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 88
  19. 2020CharacterBERT: Reconciling ELMo and BERT for Word-Level Open-Vocabulary Representations From Characterscited by 87
  20. 2020Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predictcited by 76
  21. 2020Imputer: Sequence Modelling via Imputation and Dynamic Programmingcited by 66
  22. 2019Two-Pass End-to-End Speech Recognitioncited by 64
  23. 2021Relaxing the Conditional Independence Assumption of CTC-based ASR by Conditioning on Intermediate Predictionscited by 61
  24. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  25. 2020Distilling the Knowledge of BERT for Sequence-to-Sequence ASRcited by 41
  26. 2020Incorporating BERT into Parallel Sequence Decoding with Adapterscited by 39
  27. 2020Align-Refine: Non-Autoregressive Speech Recognition via Iterative Realignmentcited by 37
  28. 2020Improved Mask-CTC for Non-Autoregressive End-to-End ASRcited by 32
  29. 2021Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERTcited by 31
  30. 2019Transformer ASR with Contextual Block Processingcited by 28
  31. 2021A Comparative Study on Non-Autoregressive Modelings for Speech-to-Text Generationcited by 28
  32. 2019Deja-vu: Double Feature Presentation and Iterated Loss in Deep Transformer Networkscited by 24
  33. 2021Innovative Bert-based Reranking Language Models for Speech Recognitioncited by 23
  34. 2022Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systemscited by 23
  35. 2021Integration of Pre-trained Networks with Continuous Token Interface for End-to-End Spoken Language Understandingcited by 22
  36. 2022Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language modelscited by 21
  37. 2018Hierarchical Multi Task Learning With CTCcited by 20
  38. 2022Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizerscited by 19

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 4; 2024: 3; 2025: 3.

11 papers.

  1. 2022BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encodercited by 8
  2. 2025Deep Learning Approaches for Multimodal Intent Recognition: A Surveycited by 6
  3. 2023Decoupled Structure for Improved Adaptability of End-to-End Modelscited by 3
  4. 2024Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recogniserscited by 3
  5. 2024Optimizing Contextual Speech Recognition Using Vector Quantization for Efficient Retrievalcited by 3
  6. 2023Multiple Representation Transfer from Large Language Models to End-to-End ASR Systemscited by 2
  7. 2023Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognitioncited by 2
  8. 2025Grapheme-Coherent Phonemic and Prosodic Annotation of Speech by Implicit and Explicit Grapheme Conditioningcited by 2
  9. 2023Generalized zero-shot audio-to-intent classificationcited by 1
  10. 2024UniEnc-CASSNAT: An Encoder-only Non-autoregressive ASR for Speech SSL Modelscited by 1
  11. 2025A Non-autoregressive Model for Joint STT and TTScited by 1