# A Simplified Fully Quantized Transformer for End-to-end Speech Recognition

arXiv: https://arxiv.org/abs/1911.03604 · HTML: https://mlsanity.com/paper/1911.03604/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Nov 2019.

## Built on by (most cited first, 11 of 11)

- [DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptation](https://mlsanity.com/paper/2210.07558/) (2022)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022)
- [Integer-only Zero-shot Quantization for Efficient Speech Recognition](https://mlsanity.com/paper/2103.16827/) (2021)
- [A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantization](https://mlsanity.com/paper/2307.12659/) (2023)
- [Training Acceleration of Low-Rank Decomposed Networks using Sequential Freezing and Rank Quantization](https://mlsanity.com/paper/2309.03824/) (2023)
- [Speeding up Resnet Architecture with Layers Targeted Low Rank Decomposition](https://mlsanity.com/paper/2309.12412/) (2023)
- [Continuation KD: Improved Knowledge Distillation through the Lens of Continuation Optimization](https://mlsanity.com/paper/2212.05998/) (2022)
- [Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model](https://mlsanity.com/paper/2406.19995/) (2024)
- [Low-bit Shift Network for End-to-End Spoken Language Understanding](https://mlsanity.com/paper/2207.07497/) (2022)
- [Accelerating the Low-Rank Decomposed Models](https://mlsanity.com/paper/2407.20266/) (2024)

## Abstract (arXiv)

While significant improvements have been made in recent years in terms of end-to-end automatic speech recognition (ASR) performance, such improvements were obtained through the use of very large neural networks, unfit for embedded use on edge devices. That being said, in this paper, we work on simplifying and compressing Transformer-based encoder-decoder architectures for the end-to-end ASR task. We empirically introduce a more compact Speech-Transformer by investigating the impact of discarding particular modules on the performance of the model. Moreover, we evaluate reducing the numerical precision of our network's weights and activations while maintaining the performance of the full-precision model. Our experiments show that we can reduce the number of parameters of the full-precision model and then further compress the model 4x by fully quantizing to 8-bit fixed point precision.