# Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer

arXiv: https://arxiv.org/abs/1910.13923 · HTML: https://mlsanity.com/paper/1910.13923/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 30 Oct 2019.

## Built on by (most cited first, 30 of 47)

- [A Survey on Efficient Inference for Large Language Models](https://mlsanity.com/paper/2404.14294/) (2024)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generation](https://mlsanity.com/paper/2203.13161/) (2022)
- [Modular Deep Learning](https://mlsanity.com/paper/2302.11529/) (2023)
- [Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Models](https://mlsanity.com/paper/2402.02244/) (2024)
- [One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesia](https://mlsanity.com/paper/2203.13357/) (2022)
- [Few-Shot Bot: Prompt-Based Learning for Dialogue Systems](https://mlsanity.com/paper/2110.08118/) (2021)
- [Audio-Driven Co-Speech Gesture Video Generation](https://mlsanity.com/paper/2212.02350/) (2022)
- [Automatic Speech Recognition using Advanced Deep Learning Approaches: A survey](https://mlsanity.com/paper/2403.01255/) (2024)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [NusaCrowd: Open Source Initiative for Indonesian NLP Resources](https://mlsanity.com/paper/2212.09648/) (2022)
- [Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalization](https://mlsanity.com/paper/2304.14535/) (2023)
- [Shrinking Bigfoot: Reducing wav2vec 2.0 footprint](https://mlsanity.com/paper/2103.15760/) (2021)
- [Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator](https://mlsanity.com/paper/2305.15099/) (2023)
- [Automatic Speech Recognition Datasets in Cantonese: A Survey and New Dataset](https://mlsanity.com/paper/2201.02419/) (2022)
- [Focus Your Attention (with Adaptive IIR Filters)](https://mlsanity.com/paper/2305.14952/) (2023)
- [CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognition](https://mlsanity.com/paper/2201.03804/) (2022)
- [On the Long Range Abilities of Transformers](https://mlsanity.com/paper/2311.16620/) (2023)
- [BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences](https://mlsanity.com/paper/2403.09347/) (2024)
- [Enabling Resource-efficient AIoT System with Cross-level Optimization: A survey](https://mlsanity.com/paper/2309.15467/) (2023)
- [An Empirical Study on Prompt Compression for Large Language Models](https://mlsanity.com/paper/2505.00019/) (2025)
- [Greenformer: Factorization Toolkit for Efficient Deep Neural Networks](https://mlsanity.com/paper/2109.06762/) (2021)
- [LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximation](https://mlsanity.com/paper/2502.20583/) (2025)
- [Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding](https://mlsanity.com/paper/2206.14318/) (2022)
- [Stable Language Model Pre-training by Reducing Embedding Variability](https://mlsanity.com/paper/2409.07787/) (2024)
- [Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximation](https://mlsanity.com/paper/2108.10808/) (2021)
- [Spectraformer: A Unified Random Feature Framework for Transformer](https://mlsanity.com/paper/2405.15310/) (2024)
- [ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers](https://mlsanity.com/paper/2310.02489/) (2023)
- [Harnessing Orthogonality to Train Low-Rank Neural Networks](https://mlsanity.com/paper/2401.08505/) (2024)
- [PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference](https://mlsanity.com/paper/2409.15104/) (2024)

## Abstract (arXiv)

Highly performing deep neural networks come at the cost of computational complexity that limits their practicality for deployment on portable devices. We propose the low-rank transformer (LRT), a memory-efficient and fast neural architecture that significantly reduces the parameters and boosts the speed of training and inference for end-to-end speech recognition. Our approach reduces the number of parameters of the network by more than 50% and speeds up the inference time by around 1.35x compared to the baseline transformer model. The experiments show that our LRT model generalizes better and yields lower error rates on both validation and test sets compared to an uncompressed transformer model. The LRT model outperforms those from existing works on several datasets in an end-to-end setting without using an external language model or acoustic data.