mlsanityData to 7 Oct 2026

Attention-Free Keyword Spotting

arXiv 2110.07749 ↗First posted 14 Oct 2021cs.LG cs.SD eess.AScited by 2

Not on the map. Why?

Abstract (arXiv)

Till now, attention-based models have been used with great success in the keyword spotting problem domain. However, in light of recent advances in deep learning, the question arises whether self-attention is truly irreplaceable for recognizing speech keywords. We thus explore the usage of gated MLPs --previously shown to be alternatives to transformers in vision tasks-- for the keyword spotting task. We provide a family of highly efficient MLP-based models for keyword spotting, with less than 0.5 million parameters. We show that our approach achieves competitive performance on Google Speech Commands V2-12 and V2-35 benchmarks with much fewer parameters than self-attention-based methods.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 14 Oct 2021.

Builds on

26 of 32 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
14 more
  1. 2021AST: Audio Spectrogram Transformercited by 634
  2. 2021Pay Attention to MLPscited by 367
  3. 2021ResMLP: Feedforward networks for image classification with data-efficient trainingcited by 338
  4. 2021Video Transformer Networkcited by 271
  5. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  6. 2017Deep Residual Learning for Small-Footprint Keyword Spottingcited by 68
  7. 2021Annealing Knowledge Distillationcited by 61
  8. 2021Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNetcited by 61
  9. 2021Keyword Transformer: A Self-Attention Model for Keyword Spottingcited by 59
  10. 2020Streaming keyword spotting on mobile devicescited by 47
  11. 2018A neural attention model for speech command recognitioncited by 44
  12. 2020MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognitioncited by 36
  13. 2021Learning Efficient Representations for Keyword Spotting with Triplet Losscited by 24
  14. 2021Compressing 1D Time-Channel Separable Convolutions using Sparse Random Ternary Matricescited by 2

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 0; 2024: 0; 2025: 1.

2 papers.

  1. 2025SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognitioncited by 2
  2. 2022Learning Audio Representations with MLPscited by 0