mlsanityData to 7 Oct 2026

Transducer Consistency Regularization for Speech to Text Applications

arXiv 2410.07491 ↗First posted 9 Oct 2024cs.CL eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Consistency regularization is a commonly used practice to encourage the model to generate consistent representation from distorted input features and improve model generalization. It shows significant improvement on various speech applications that are optimized with cross entropy criterion. However, it is not straightforward to apply consistency regularization for the transducer-based approaches, which are widely adopted for speech applications due to the competitive performance and streaming characteristic. The main challenge is from the vast alignment space of the transducer optimization criterion and not all the alignments within the space contribute to the model optimization equally. In this study, we present Transducer Consistency Regularization (TCR), a consistency regularization method for transducer models. We apply distortions such as spec augmentation and dropout to create different data views and minimize the distribution difference. We utilize occupational probabilities to give different weights on transducer output distributions, thus only alignments close to oracle alignments would contribute to the model learning. Our experiments show the proposed method is superior to other consistency regularization implementations and could effectively reduce word error rate (WER) by 4.3% relatively comparing with a strong baseline on the textscLibrispeech dataset.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 39 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

27 of 39 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  5. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  6. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  7. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  8. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  9. 2020Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Losscited by 232
  10. 2019NeMo: a toolkit for building AI applications using Neural Modulescited by 230
  11. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  12. 2021Recent Advances in End-to-End Automatic Speech Recognitioncited by 138
15 more
  1. 2020Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognitioncited by 111
  2. 2021Intermediate Loss Regularization for CTC-based Speech Recognitioncited by 95
  3. 2023Zipformer: A faster and better encoder for automatic speech recognitioncited by 77
  4. 2019Transformer-Transducer: End-to-End Speech Recognition with Self-Attentioncited by 68
  5. 2020On the Comparison of Popular End-to-End Models for Large Scale Speech Recognitioncited by 62
  6. 2022Pruned RNN-T for fast, memory-efficient ASR trainingcited by 46
  7. 2019A comparison of end-to-end models for long-form speech recognitioncited by 42
  8. 2020Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memorycited by 35
  9. 2022Large-Scale Streaming End-to-End Speech Translation with Neural Transducerscited by 31
  10. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 29
  11. 2023Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Taskscited by 20
  12. 2020A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognitioncited by 8
  13. 2023Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Predictioncited by 7
  14. 2023An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translationcited by 6
  15. 2022SAN: a robust end-to-end ASR model architecturecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularizationcited by 1