# Accelerator-Aware Training for Transducer-Based Speech Recognition

arXiv: https://arxiv.org/abs/2305.07778 · HTML: https://mlsanity.com/paper/2305.07778/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 27 references that are in the map.
- First posted on arXiv 12 May 2023.

## Builds on (most cited first, 20 of 20)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [Transformer-based Acoustic Modeling for Hybrid Speech Recognition](https://mlsanity.com/paper/1910.09799/) (2019)
- [FastEmit: Low-latency Streaming ASR with Sequence-level Emission Regularization](https://mlsanity.com/paper/2010.11148/) (2020)
- [Simple Recurrent Units for Highly Parallelizable Recurrence](https://mlsanity.com/paper/1709.02755/) (2017)
- [Efficient Knowledge Distillation for RNN-Transducer Models](https://mlsanity.com/paper/2011.06110/) (2020)
- [Optimizing Speech Recognition For The Edge](https://mlsanity.com/paper/1909.12408/) (2019)
- [Exploring Transformers for Large-Scale Speech Recognition](https://mlsanity.com/paper/2005.09684/) (2020)
- [Accelerator-Aware Pruning for Convolutional Neural Networks](https://mlsanity.com/paper/1804.09862/) (2018)
- [On the efficient representation and execution of deep acoustic models](https://mlsanity.com/paper/1607.04683/) (2016)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Amortized Neural Networks for Low-Latency Speech Recognition](https://mlsanity.com/paper/2108.01553/) (2021)
- [Sparsification via Compressed Sensing for Automatic Speech Recognition](https://mlsanity.com/paper/2102.04932/) (2021)
- [ConvRNN-T: Convolutional Augmented Recurrent Neural Network Transducers for Streaming Speech Recognition](https://mlsanity.com/paper/2209.14868/) (2022)
- [On the quantization of recurrent neural networks](https://mlsanity.com/paper/2101.05453/) (2021)
- [Collaborative Training of Acoustic Encoders for Speech Recognition](https://mlsanity.com/paper/2106.08960/) (2021)
- [Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimization](https://mlsanity.com/paper/2108.01704/) (2021)

## Abstract (arXiv)

Machine learning model weights and activations are represented in full-precision during training. This leads to performance degradation in runtime when deployed on neural network accelerator (NNA) chips, which leverage highly parallelized fixed-point arithmetic to improve runtime memory and latency. In this work, we replicate the NNA operators during the training phase, accounting for the degradation due to low-precision inference on the NNA in back-propagation. Our proposed method efficiently emulates NNA operations, thus foregoing the need to transfer quantization error-prone data to the Central Processing Unit (CPU), ultimately reducing the user perceived latency (UPL). We apply our approach to Recurrent Neural Network-Transducer (RNN-T), an attractive architecture for on-device streaming speech recognition tasks. We train and evaluate models on 270K hours of English data and show a 5-7% improvement in engine latency while saving up to 10% relative degradation in WER.