# Transducer Consistency Regularization for Speech to Text Applications

arXiv: https://arxiv.org/abs/2410.07491 · HTML: https://mlsanity.com/paper/2410.07491/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 39 references that are in the map.
- First posted on arXiv 9 Oct 2024.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing](https://mlsanity.com/paper/1808.06226/) (2018)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning](https://mlsanity.com/paper/1704.03976/) (2017)
- [NeMo: a toolkit for building AI applications using Neural Modules](https://mlsanity.com/paper/1909.09577/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [R-Drop: Regularized Dropout for Neural Networks](https://mlsanity.com/paper/2106.14448/) (2021)
- [Recent Advances in End-to-End Automatic Speech Recognition](https://mlsanity.com/paper/2111.01690/) (2021)
- [Emformer: Efficient Memory Transformer Based Acoustic Model For Low Latency Streaming Speech Recognition](https://mlsanity.com/paper/2010.10759/) (2020)
- [Intermediate Loss Regularization for CTC-based Speech Recognition](https://mlsanity.com/paper/2102.03216/) (2021)
- [Zipformer: A faster and better encoder for automatic speech recognition](https://mlsanity.com/paper/2310.11230/) (2023)
- [Transformer-Transducer: End-to-End Speech Recognition with Self-Attention](https://mlsanity.com/paper/1910.12977/) (2019)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Pruned RNN-T for fast, memory-efficient ASR training](https://mlsanity.com/paper/2206.13236/) (2022)
- [A comparison of end-to-end models for long-form speech recognition](https://mlsanity.com/paper/1911.02242/) (2019)
- [Efficient Knowledge Distillation for RNN-Transducer Models](https://mlsanity.com/paper/2011.06110/) (2020)
- [Streaming Transformer-based Acoustic Models Using Self-attention with Augmented Memory](https://mlsanity.com/paper/2005.08042/) (2020)
- [Large-Scale Streaming End-to-End Speech Translation with Neural Transducers](https://mlsanity.com/paper/2204.05352/) (2022)
- [Hybrid Transducer and Attention based Encoder-Decoder Modeling for Speech-to-Text Tasks](https://mlsanity.com/paper/2305.03101/) (2023)
- [A Comparison of Label-Synchronous and Frame-Synchronous End-to-End Models for Speech Recognition](https://mlsanity.com/paper/2005.10113/) (2020)
- [Transduce and Speak: Neural Transducer for Text-to-Speech with Semantic Token Prediction](https://mlsanity.com/paper/2311.02898/) (2023)
- [An Empirical Study of Consistency Regularization for End-to-End Speech-to-Text Translation](https://mlsanity.com/paper/2308.14482/) (2023)
- [SAN: a robust end-to-end ASR model architecture](https://mlsanity.com/paper/2210.15285/) (2022)

## Built on by (most cited first, 1 of 1)

- [Reducing the Offline-Streaming Gap for Unified ASR Transducer with Consistency Regularization](https://mlsanity.com/paper/2604.19079/) (2026)

## Abstract (arXiv)

Consistency regularization is a commonly used practice to encourage the model to generate consistent representation from distorted input features and improve model generalization. It shows significant improvement on various speech applications that are optimized with cross entropy criterion. However, it is not straightforward to apply consistency regularization for the transducer-based approaches, which are widely adopted for speech applications due to the competitive performance and streaming characteristic. The main challenge is from the vast alignment space of the transducer optimization criterion and not all the alignments within the space contribute to the model optimization equally. In this study, we present Transducer Consistency Regularization (TCR), a consistency regularization method for transducer models. We apply distortions such as spec augmentation and dropout to create different data views and minimize the distribution difference. We utilize occupational probabilities to give different weights on transducer output distributions, thus only alignments close to oracle alignments would contribute to the model learning. Our experiments show the proposed method is superior to other consistency regularization implementations and could effectively reduce word error rate (WER) by 4.3% relatively comparing with a strong baseline on the textscLibrispeech dataset.