mlsanityData to 14 Oct 2026

Bottleneck Low-rank Transformers for Low-resource Spoken Language Understanding

arXiv 2206.14318 ↗First posted 28 Jun 2022cs.CL cs.LG eess.AScited by 6

Not on the map. Why?

Abstract (arXiv)

End-to-end spoken language understanding (SLU) systems benefit from pretraining on large corpora, followed by fine-tuning on application-specific data. The resulting models are too large for on-edge applications. For instance, BERT-based systems contain over 110M parameters. Observing the model is overparameterized, we propose lean transformer structure where the dimension of the attention mechanism is automatically reduced using group sparsity. We propose a variant where the learned attention subspace is transferred to an attention bottleneck layer. In a low-resource setting and without pre-training, the resulting compact SLU model achieves accuracies competitive with pre-trained large models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 25 of its 33 references that are in the map. First posted on arXiv 28 Jun 2022.

Builds on

25 of 33 references.

References · 25

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  5. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 377
  6. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 193
  7. 2018Towards end-to-end spoken language understandingcited by 92
  8. 2020Low-Rank Bottleneck in Multi-head Attention Modelscited by 89
  9. 2019Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformercited by 47
  10. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  11. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  12. 2020End-to-End Neural Transformer Based Spoken Language Understandingcited by 31
13 more
  1. 2020Exploring Transfer Learning For End-to-End Spoken Language Understandingcited by 17
  2. 2021Knowledge Distillation from BERT Transformer to Speech Transformer for Intent Classificationcited by 14
  3. 2022Spectro-Temporal Deep Features for Disordered Speech Assessment and Recognitioncited by 13
  4. 2018Capsule Networks for Low Resource Spoken Language Understandingcited by 11
  5. 2021Speak or Chat with Me: End-to-End Spoken Language Understanding System with Flexible Inputscited by 8
  6. 2021Sequential End-to-End Intent and Slot Label Classification and Localizationcited by 7
  7. 2017Just ASK: Building an Architecture for Extensible Self-Service Spoken Language Understandingcited by 6
  8. 2021Pre-training for Spoken Language Understanding with Joint Textual and Phonetic Representation Learningcited by 6
  9. 2022Insights on Neural Representations for End-to-End Speech Recognitioncited by 5
  10. 2021A Study into Pre-training Strategies for Spoken Language Understanding on Dysarthric Speechcited by 4
  11. 2019Low Rank Factorization for Compact Multi-Head Self-Attentioncited by 3
  12. 2022Three-Module Modeling For End-to-End Spoken Language Understanding Using Pre-trained DNN-HMM-Based Acoustic-Phonetic Modelcited by 3
  13. 2020A light transformer for speech-to-intent applicationscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 3; 2026: 1.

6 papers.

  1. 2023Modular Deep Learningcited by 83
  2. 2025SSVD: Structured SVD for Parameter-Efficient Fine-Tuning and Benchmarking under Domain Shift in ASRcited by 4
  3. 2025Input Conditioned Layer Dropping in Speech Foundation Modelscited by 2
  4. 2024The Computational Learning of Construction Grammars: State of the Art and Prospective Roadmapcited by 1
  5. 2025An Effective Training Framework for Light-Weight Automatic Speech Recognition Modelscited by 1
  6. 2026AURA: Uncertainty-Routed Activation Editing for Acoustic Grounding in Speech Foundation Modelscited by 0