# BECTRA: Transducer-based End-to-End ASR with BERT-Enhanced Encoder arXiv: https://arxiv.org/abs/2211.00792 ยท HTML: https://mlsanity.com/paper/2211.00792/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 41 references that are in the map. - First posted on arXiv 2 Nov 2022. ## Builds on (most cited first, 30 of 32) - [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017) - [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020) - [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020) - [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019) - [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019) - [Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates](https://mlsanity.com/paper/1804.10959/) (2018) - [ESPnet: End-to-End Speech Processing Toolkit](https://mlsanity.com/paper/1804.00015/) (2018) - [AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baseline](https://mlsanity.com/paper/1709.05522/) (2017) - [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021) - [Mask CTC: Non-Autoregressive End-to-End ASR with CTC and Mask Predict](https://mlsanity.com/paper/2005.08700/) (2020) - [Imputer: Sequence Modelling via Imputation and Dynamic Programming](https://mlsanity.com/paper/2002.08926/) (2020) - [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019) - [Distilling the Knowledge of BERT for Sequence-to-Sequence ASR](https://mlsanity.com/paper/2008.03822/) (2020) - [Improved Mask-CTC for Non-Autoregressive End-to-End ASR](https://mlsanity.com/paper/2010.13270/) (2020) - [Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT](https://mlsanity.com/paper/2102.07594/) (2021) - [Innovative Bert-based Reranking Language Models for Speech Recognition](https://mlsanity.com/paper/2104.04950/) (2021) - [Effect and Analysis of Large-scale Language Model Rescoring on Competitive ASR Systems](https://mlsanity.com/paper/2204.00212/) (2022) - [Improving non-autoregressive end-to-end speech recognition with pre-trained acoustic and language models](https://mlsanity.com/paper/2201.10103/) (2022) - [Knowledge Transfer from Large-scale Pretrained Language Models to End-to-end Speech Recognizers](https://mlsanity.com/paper/2202.07894/) (2022) - [Efficiently Fusing Pretrained Acoustic and Linguistic Encoders for Low-resource Speech Recognition](https://mlsanity.com/paper/2101.06699/) (2021) - [Wav-BERT: Cooperative Acoustic and Linguistic Representation Learning for Low-Resource Speech Recognition](https://mlsanity.com/paper/2109.09161/) (2021) - [ASR Rescoring and Confidence Estimation with ELECTRA](https://mlsanity.com/paper/2110.01857/) (2021) - [Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Model](https://mlsanity.com/paper/2112.07254/) (2021) - [A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategies](https://mlsanity.com/paper/2201.05420/) (2022) - [Non-autoregressive Transformer-based End-to-end ASR using BERT](https://mlsanity.com/paper/2104.04805/) (2021) - [A context-aware knowledge transferring strategy for CTC-based ASR](https://mlsanity.com/paper/2210.06244/) (2022) - [Speech Recognition by Simply Fine-tuning BERT](https://mlsanity.com/paper/2102.00291/) (2021) - [Streaming End-to-End ASR based on Blockwise Non-Autoregressive Models](https://mlsanity.com/paper/2107.09428/) (2021) ## Built on by (most cited first, 8 of 8) - [Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR](https://mlsanity.com/paper/2409.02239/) (2024) - [Multiple Representation Transfer from Large Language Models to End-to-End ASR Systems](https://mlsanity.com/paper/2309.04031/) (2023) - [Harnessing the Zero-Shot Power of Instruction-Tuned Large Language Model in End-to-End Speech Recognition](https://mlsanity.com/paper/2309.10524/) (2023) - [A Non-autoregressive Model for Joint STT and TTS](https://mlsanity.com/paper/2501.09104/) (2025) - [Cross-modal Knowledge Transfer Learning as Graph Matching Based on Optimal Transport for ASR](https://mlsanity.com/paper/2505.13079/) (2025) - [Bidirectional Representations for Low Resource Spoken Language Understanding](https://mlsanity.com/paper/2211.14320/) (2022) - [New Insights into Optimal Alignment of Acoustic and Linguistic Representations for Knowledge Transfer in ASR](https://mlsanity.com/paper/2509.05609/) (2025) - [Optimal transport meets speech: a tutorial review](https://mlsanity.com/paper/2609.31787/) (2026) ## Abstract (arXiv) We present BERT-CTC-Transducer (BECTRA), a novel end-to-end automatic speech recognition (E2E-ASR) model formulated by the transducer with a BERT-enhanced encoder. Integrating a large-scale pre-trained language model (LM) into E2E-ASR has been actively studied, aiming to utilize versatile linguistic knowledge for generating accurate text. One crucial factor that makes this integration challenging lies in the vocabulary mismatch; the vocabulary constructed for a pre-trained LM is generally too large for E2E-ASR training and is likely to have a mismatch against a target ASR domain. To overcome such an issue, we propose BECTRA, an extended version of our previous BERT-CTC, that realizes BERT-based E2E-ASR using a vocabulary of interest. BECTRA is a transducer-based model, which adopts BERT-CTC for its encoder and trains an ASR-specific decoder using a vocabulary suitable for a target task. With the combination of the transducer and BERT-CTC, we also propose a novel inference algorithm for taking advantage of both autoregressive and non-autoregressive decoding. Experimental results on several ASR tasks, varying in amounts of data, speaking styles, and languages, demonstrate that BECTRA outperforms BERT-CTC by effectively dealing with the vocabulary mismatch while exploiting BERT knowledge.