mlsanityData to 14 Oct 2026

Enhanced Hybrid Transducer and Attention Encoder Decoder with Text Data

arXiv 2506.19159 ↗First posted 23 Jun 2025cs.CL cs.SD eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

A joint speech and text optimization method is proposed for hybrid transducer and attention-based encoder decoder (TAED) modeling to leverage large amounts of text corpus and enhance ASR accuracy. The joint TAED (J-TAED) is trained with both speech and text input modalities together, while it only takes speech data as input during inference. The trained model can unify the internal representations from different modalities, and be further extended to text-based domain adaptation. It can effectively alleviate data scarcity for mismatch domain tasks since no speech data is required. Our experiments show J-TAED successfully integrates speech and linguistic information into one model, and reduce the WER by 5.8 ~12.8% on the Librispeech dataset. The model is also evaluated on two out-of-domain datasets: one is finance and another is named entity focused. The text-based domain adaptation brings 15.3% and 17.8% WER reduction on those two datasets respectively.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 35 of its 38 references that are in the map. First posted on arXiv 23 Jun 2025.

Builds on

35 of 38 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2016Layer Normalizationcited by 5,669
  3. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  7. 2018Self-Attention with Relative Position Representationscited by 1,120
  8. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  9. 2019On the Relationship between Self-Attention and Convolutional Layerscited by 300
  10. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 235
  11. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 235
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 140
23 more
  1. 2023Prompting Large Language Models with Speech Recognition Abilitiescited by 140
  2. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 96
  3. 2022MAESTRO: Matched Speech Text Representations through Modality Matchingcited by 79
  4. 2021SPGISpeech: 5,000 hours of transcribed financial audio for fully formatted end-to-end speech recognitioncited by 78
  5. 2023SALM: Speech-augmented Language Model with In-context Learning for Speech Recognition and Translationcited by 76
  6. 2021SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Trainingcited by 69
  7. 2019Investigating Multilingual NMT Representations at Scalecited by 68
  8. 2019Two-Pass End-to-End Speech Recognitioncited by 64
  9. 2022Unified Speech-Text Pre-training for Speech Translation and Recognitioncited by 64
  10. 2021Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Taskcited by 54
  11. 2020A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Taskscited by 51
  12. 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
  13. 2022SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-trainingcited by 40
  14. 2022CTC Alignments Improve Autoregressive Translationcited by 25
  15. 2022Integrating Text Inputs For Training and Adapting RNN Transducer ASR Modelscited by 20
  16. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  17. 2018Learning pronunciation from a foreign language in speech synthesis networkscited by 12
  18. 2022Fast and accurate factorized neural transducer for text adaption of end-to-end speech recognition modelscited by 11
  19. 20224D ASR: Joint modeling of CTC, Attention, Transducer, and Mask-Predict decoderscited by 7
  20. 2023Text-only Domain Adaptation using Unified Speech-Text Representation in Transducercited by 5
  21. 2023t-SOT FNT: Streaming Multi-talker ASR with Text-only Domain Adaptation Capabilitycited by 4
  22. 2024Effective internal language model training and fusion for factorized transducer modelcited by 3
  23. 2023Improved Factorized Neural Transducer Model For text-only Domain Adaptationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TRADE: Transducer-Augmented Decoder for Speech LLMcited by 0