mlsanityData to 7 Oct 2026

Decoupled Structure for Improved Adaptability of End-to-End Models

arXiv 2308.13345 ↗First posted 25 Aug 2023eess.AS cs.CL cs.SDcited by 3

Not on the map. Why?

Abstract (arXiv)

Although end-to-end (E2E) trainable automatic speech recognition (ASR) has shown great success by jointly learning acoustic and linguistic information, it still suffers from the effect of domain shifts, thus limiting potential applications. The E2E ASR model implicitly learns an internal language model (LM) which characterises the training distribution of the source domain, and the E2E trainable nature makes the internal LM difficult to adapt to the target domain with text-only data To solve this problem, this paper proposes decoupled structures for attention-based encoder-decoder (Decoupled-AED) and neural transducer (Decoupled-Transducer) models, which can achieve flexible domain adaptation in both offline and online scenarios while maintaining robust intra-domain performance. To this end, the acoustic and linguistic parts of the E2E model decoder (or prediction network) are decoupled, making the linguistic component (i.e. internal LM) replaceable. When encountering a domain shift, the internal LM can be directly replaced during inference by a target-domain LM, without re-training or using domain-specific paired speech-text data. Experiments for E2E ASR models trained on the LibriSpeech-100h corpus showed that the proposed decoupled structure gave 15.1% and 17.2% relative word error rate reductions on the TED-LIUM 2 and AESRC2020 corpora while still maintaining performance on intra-domain data.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 49 of its 75 references that are in the map. First posted on arXiv 25 Aug 2023.

Builds on

49 of 75 references.

References · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  3. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  4. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  5. 2018ESPnet: End-to-End Speech Processing Toolkitcited by 554
  6. 2015Listen, Attend and Spellcited by 542
  7. 2020Unsupervised Cross-lingual Representation Learning for Speech Recognitioncited by 496
  8. 2015Attention-Based Models for Speech Recognitioncited by 400
  9. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  10. 2021SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processingcited by 177
  11. 2015On Using Monolingual Corpora in Neural Machine Translationcited by 149
  12. 2021Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Trainingcited by 148
37 more
  1. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
  2. 2019Europarl-ST: A Multilingual Corpus For Speech Translation Of Parliamentary Debatescited by 114
  3. 2017An analysis of incorporating an external language model into a sequence-to-sequence modelcited by 109
  4. 2020Hybrid Autoregressive Transducer (hat)cited by 106
  5. 2017Cold Fusion: Training Seq2Seq Models Together with Language Modelscited by 87
  6. 2020ESPnet-ST: All-in-One Speech Translation Toolkitcited by 82
  7. 2020Internal Language Model Estimation for Domain-Adaptive End-to-End Speech Recognitioncited by 80
  8. 2020A Density Ratio Approach to Language Model Fusion in End-To-End Automatic Speech Recognitioncited by 73
  9. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  10. 2020Using Synthetic Audio to Improve The Recognition of Out-Of-Vocabulary Words in End-To-End ASR Systemscited by 41
  11. 2021Internal Language Model Training for Domain-Adaptive End-to-End Speech Recognitioncited by 37
  12. 2021Factorized Neural Transducer for Efficient Language Model Adaptationcited by 31
  13. 2021Investigating Methods to Improve Language Model Integration for Attention-based Encoder-Decoder ASR Modelscited by 25
  14. 2021The Accented English Speech Recognition Challenge 2020: Open Datasets, Tracks, Baselines, Results and Methodscited by 24
  15. 2021Searchable Hidden Intermediates for End-to-End Models of Decomposable Sequence Taskscited by 24
  16. 2021Investigating the Reordering Capability in CTC-based Non-Autoregressive End-to-End Speech Translationcited by 23
  17. 2019Improving Performance of End-to-End ASR on Numeric Sequencescited by 17
  18. 2021Internal Language Model Adaptation with Text-Only Data for End-to-End Speech Recognitioncited by 17
  19. 2021On Language Model Integration for RNN Transducer based Speech Recognitioncited by 17
  20. 2022Modular Hybrid Autoregressive Transducercited by 17
  21. 2022A Study of Transducer based End-to-End ASR with ESPnet: Architecture, Auxiliary Loss and Decoding Strategiescited by 15
  22. 2021Improving Hybrid CTC/Attention End-to-end Speech Recognition with Pretrained Acoustic and Language Modelcited by 14
  23. 2022Pre-Training Transformer Decoder for End-to-End ASR Model with Unpaired Speech Datacited by 12
  24. 2022Non-Parametric Domain Adaptation for End-to-End Speech Translationcited by 12
  25. 2022Blockwise Streaming Transformer for Spoken Language Understanding and Simultaneous Speech Translationcited by 11
  26. 2022BERT Meets CTC: New Formulation of End-to-End Speech Recognition with Pre-trained Masked Language Modelcited by 11
  27. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
  28. 2021Improving Streaming Transformer Based ASR Under a Framework of Self-supervised Learningcited by 10
  29. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Modelscited by 10
  30. 2021Fast-MD: Fast Multi-Decoder End-to-End Speech Translation with Non-Autoregressive Hidden Intermediatescited by 9
  31. 2019Integrating Source-channel and Attention-based Sequence-to-sequence Models for Speech Recognitioncited by 8
  32. 2023Adaptable End-to-End ASR Models using Replaceable Internal LMs and Residual Softmaxcited by 8
  33. 2022UFO2: A unified pre-training framework for online and offline speech recognitioncited by 7
  34. 2022A Likelihood Ratio based Domain Adaptation Method for E2E Modelscited by 5
  35. 2022Residual Language Model for End-to-end Speech Recognitioncited by 5
  36. 2021Improving Pseudo-label Training For End-to-end Speech Recognition Using Gradient Maskcited by 3
  37. 2022Ask2Mask: Guided Data Selection for Masked Speech Modelingcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 1; 2025: 1.

3 papers.

  1. 2024Transducer-Llama: Integrating LLMs into Streamable Transducer-based Speech Recognitioncited by 4
  2. 2023Hybrid Attention-based Encoder-decoder Model for Efficient Language Model Adaptationcited by 1
  3. 2025WhisTLE: Deeply Supervised, Text-Only Domain Adaptation for Small Pretrained Speech Recognition Transformerscited by 0