# On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition

arXiv: https://arxiv.org/abs/2312.09842 · HTML: https://mlsanity.com/paper/2312.09842/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 22 of its 28 references that are in the map.
- First posted on arXiv 15 Dec 2023.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [Mixed Precision Training](https://mlsanity.com/paper/1710.03740/) (2017)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [Hybrid Autoregressive Transducer (hat)](https://mlsanity.com/paper/2003.07705/) (2020)
- [Two-Pass End-to-End Speech Recognition](https://mlsanity.com/paper/1908.10992/) (2019)
- [Cascaded encoders for unifying streaming and non-streaming ASR](https://mlsanity.com/paper/2010.14606/) (2020)
- [Efficient Knowledge Distillation for RNN-Transducer Models](https://mlsanity.com/paper/2011.06110/) (2020)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [Tiny Transducer: A Highly-efficient Speech Recognition Model on Edge Devices](https://mlsanity.com/paper/2101.06856/) (2021)
- [A review of on-device fully neural end-to-end automatic speech recognition algorithms](https://mlsanity.com/paper/2012.07974/) (2020)
- [Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition](https://mlsanity.com/paper/2210.00169/) (2022)
- [Improved Multi-Stage Training of Online Attention-based Encoder-Decoder Models](https://mlsanity.com/paper/1912.12384/) (2019)
- [Two-Pass End-to-End ASR Model Compression](https://mlsanity.com/paper/2201.02741/) (2022)
- [Improving RNN transducer with normalized jointer network](https://mlsanity.com/paper/2011.01576/) (2020)
- [Practical Conformer: Optimizing size, speed and flops of Conformer for on-Device and cloud ASR](https://mlsanity.com/paper/2304.00171/) (2023)

## Built on by (most cited first, 1 of 1)

- [Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models](https://mlsanity.com/paper/2502.05837/) (2025)

## Abstract (arXiv)

Recently, the cascaded two-pass architecture has emerged as a strong contender for on-device automatic speech recognition (ASR). A cascade of causal and shallow non-causal encoders coupled with a shared decoder enables operation in both streaming and look-ahead modes. In this paper, we propose shallow cascaded model by combining various model compression techniques such as knowledge distillation, shared decoder, and tied-and-reduced transducer network in order to reduce the model footprint. The shared decoder is changed into a tied-and-reduced network. The cascaded two-pass model is further compressed using knowledge distillation using a Kullback-Leibler divergence loss on the model posteriors. We demonstrate a 50% reduction in the size of a 41 M parameter cascaded teacher model with no noticeable degradation in ASR accuracy and a 30% reduction in latency