mlsanityData to 14 Oct 2026

Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer

arXiv 1910.13923 ↗First posted 30 Oct 2019cs.CL cs.LG cs.SDcited by 47

Not on the map. Why?

Abstract (arXiv)

Highly performing deep neural networks come at the cost of computational complexity that limits their practicality for deployment on portable devices. We propose the low-rank transformer (LRT), a memory-efficient and fast neural architecture that significantly reduces the parameters and boosts the speed of training and inference for end-to-end speech recognition. Our approach reduces the number of parameters of the network by more than 50% and speeds up the inference time by around 1.35x compared to the baseline transformer model. The experiments show that our LRT model generalizes better and yields lower error rates on both validation and test sets compared to an uncompressed transformer model. The LRT model outperforms those from existing works on several datasets in an end-to-end setting without using an external language model or acoustic data.

Key facts. Cited by 47 papers in the map (data through 14 Oct 2026). First posted on arXiv 30 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 4; 2022: 10; 2023: 12; 2024: 14; 2025: 6; 2026: 1.

47 papers.

Sort:

By area:

  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  2. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 204
  3. 2022Learning Hierarchical Cross-Modal Association for Co-Speech Gesture Generationcited by 107
  4. 2023Modular Deep Learningcited by 83
  5. 2024Beyond the Limits: A Survey of Techniques to Extend the Context Length in Large Language Modelscited by 76
  6. 2022One Country, 700+ Languages: NLP Challenges for Underrepresented Languages and Dialects in Indonesiacited by 66
  7. 2021Few-Shot Bot: Prompt-Based Learning for Dialogue Systemscited by 64
  8. 2022Audio-Driven Co-Speech Gesture Video Generationcited by 55
  9. 2024Automatic Speech Recognition using Advanced Deep Learning Approaches: A surveycited by 50
  10. 2023Transformers in Speech Processing: A Surveycited by 39
  11. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  12. 2023Deep Transfer Learning for Automatic Speech Recognition: Towards Better Generalizationcited by 28
35 more
  1. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 21
  2. 2023Fourier Transformer: Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operatorcited by 20
  3. 2022Automatic Speech Recognition Datasets in Cantonese: A Survey and New Datasetcited by 12
  4. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  5. 2022CI-AVSR: A Cantonese Audio-Visual Speech Dataset for In-car Command Recognitioncited by 10
  6. 2023On the Long Range Abilities of Transformerscited by 10
  7. 2024BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequencescited by 10
  8. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  9. 2025An Empirical Study on Prompt Compression for Large Language Modelscited by 8
  10. 2021Greenformer: Factorization Toolkit for Efficient Deep Neural Networkscited by 7
  11. 2025LiteASR: Efficient Automatic Speech Recognition with Low-Rank Approximationcited by 7
  12. 2022Bottleneck Low-rank Transformers for Low-resource Spoken Language Understandingcited by 6
  13. 2024Stable Language Model Pre-training by Reducing Embedding Variabilitycited by 6
  14. 2021Greenformers: Improving Computation and Memory Efficiency in Transformer Models via Low-Rank Approximationcited by 5
  15. 2024Spectraformer: A Unified Random Feature Framework for Transformercited by 5
  16. 2023ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layerscited by 4
  17. 2024Harnessing Orthogonality to Train Low-Rank Neural Networkscited by 4
  18. 2024PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inferencecited by 4
  19. 2023FLORA: Fine-grained Low-Rank Architecture Search for Vision Transformercited by 3
  20. 2024Full-Rank No More: Low-Rank Weight Training for Modern Speech Recognition Modelscited by 3
  21. 2022On the Effectiveness of Pinyin-Character Dual-Decoding for End-to-End Mandarin Chinese ASRcited by 2
  22. 2024S3Attention: Improving Long Sequence Attention with Smoothed Skeleton Sketchingcited by 2
  23. 2025Input Conditioned Layer Dropping in Speech Foundation Modelscited by 2
  24. 2025Sliced ReLU attention: Quasi-linear contextual expressivity via sortingcited by 2
  25. 2022How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modelingcited by 1
  26. 2023Cross-Lingual Cross-Age Group Adaptation for Low-Resource Elderly Speech Emotion Recognitioncited by 1
  27. 2024PCToolkit: A Unified Plug-and-Play Prompt Compression Toolkit of Large Language Modelscited by 1
  28. 2024Long Sequence Modeling with Attention Tensorization: From Sequence to Tensor Learningcited by 1
  29. 2025An Effective Training Framework for Light-Weight Automatic Speech Recognition Modelscited by 1
  30. 2025On Information Geometry and Iterative Optimization in Model Compression: Operator Factorizationcited by 1
  31. 2022Deep Sparse Conformer for Speech Recognitioncited by 0
  32. 2023Beyond Universal Transformer: block reusing with adaptor in Transformer for automatic speech recognitioncited by 0
  33. 2024AB-Training: A Communication-Efficient Approach for Distributed Low-Rank Learningcited by 0
  34. 2024AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformerscited by 0
  35. 2026An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attentioncited by 0