mlsanityData to 7 Oct 2026

Keyword Transformer: A Self-Attention Model for Keyword Spotting

arXiv 2104.00769 ↗First posted 1 Apr 2021eess.AS cs.CL cs.LGcited by 59

Not on the map. Why?

Abstract (arXiv)

The Transformer architecture has been successful across many domains, including natural language processing, computer vision and speech recognition. In keyword spotting, self-attention has primarily been used on top of convolutional or recurrent encoders. We investigate a range of ways to adapt the Transformer architecture to keyword spotting and introduce the Keyword Transformer (KWT), a fully self-attentional architecture that exceeds state-of-the-art performance across multiple tasks without any pre-training or additional data. Surprisingly, this simple architecture outperforms more complex models that mix convolutional, recurrent and attentive layers. KWT can be used as a drop-in replacement for these models, setting two new benchmark records on the Google Speech Commands dataset with 98.6% and 97.7% accuracy on the 12 and 35-command tasks respectively.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 31 of its 35 references that are in the map. First posted on arXiv 1 Apr 2021.

Builds on

31 of 35 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2020End-to-End Object Detection with Transformerscited by 7,384
  6. 2016Layer Normalizationcited by 5,579
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  9. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
19 more
  1. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  2. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  3. 2021Video Transformer Networkcited by 271
  4. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  5. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  6. 2019Transformers without Tears: Improving the Normalization of Self-Attentioncited by 153
  7. 2017Hello Edge: Keyword Spotting on Microcontrollerscited by 108
  8. 2020Developing Real-time Streaming Transformer Transducer for Speech Recognition on Large-scale Datasetcited by 94
  9. 2021Colorization Transformercited by 89
  10. 2019Temporal Convolution for Real-time Keyword Spotting on Mobile Devicescited by 48
  11. 2020Streaming keyword spotting on mobile devicescited by 47
  12. 2018A neural attention model for speech command recognitioncited by 44
  13. 2020MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognitioncited by 36
  14. 2021Learning Efficient Representations for Keyword Spotting with Triplet Losscited by 24
  15. 2020Training Keyword Spotters with Limited and Synthesized Speech Datacited by 20
  16. 2021baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotemporal Modelingcited by 14
  17. 2021Wake Word Detection with Streaming Transformerscited by 10
  18. 2020Hybrid Transformer/CTC Networks for Hardware Efficient Voice Triggeringcited by 8
  19. 2019Ternary Hybrid Neural-Tree Networks for Highly Constrained IoT Applicationscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 16; 2023: 10; 2024: 8; 2025: 10; 2026: 5.

59 papers.

Sort:

By area:

  1. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  2. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  3. 2021Deep Spoken Keyword Spotting: An Overviewcited by 43
  4. 2023Towards Stealthy Backdoor Attacks against Speech Recognition via Elements of Soundcited by 28
  5. 2021Action Transformer: A Self-Attention Model for Short-Time Pose-Based Human Action Recognitioncited by 24
  6. 2023Leveraging Large Language Models for Exploiting ASR Uncertaintycited by 15
  7. 2023The taste of IPA: Towards open-vocabulary keyword spotting and forced alignment in any languagecited by 14
  8. 2021Temporal Knowledge Distillation for On-device Audio Classificationcited by 13
  9. 2023CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Losscited by 12
  10. 2021Visual Keyword Spotting with Attentioncited by 11
  11. 2024LMUFormer: Low Complexity Yet Powerful Spiking Model With Legendre Memory Unitscited by 11
  12. 2022Points to Patches: Enabling the Use of Self-Attention for 3D Shape Recognitioncited by 9
38 more
  1. 2021Attention mechanisms and deep learning for machine vision: A survey of the state of the artcited by 8
  2. 2022Rainbow Keywords: Efficient Incremental Learning for Online Spoken Keyword Spottingcited by 8
  3. 2023Open-vocabulary Keyword-spotting with Adaptive Instance Normalizationcited by 8
  4. 2023PQA: Exploring the Potential of Product Quantization in DNN Hardware Accelerationcited by 6
  5. 2021Marvin: an Innovative Omni-Directional Robotic Assistant for Domestic Environmentscited by 5
  6. 2022BiFSMN: Binary Neural Network for Keyword Spottingcited by 5
  7. 2024Imperceptible Rhythm Backdoor Attacks: Exploring Rhythm Transformation for Embedding Undetectable Vulnerabilities on Speech Recognitioncited by 5
  8. 2021Audiomer: A Convolutional Transformer For Keyword Spottingcited by 4
  9. 2022Avoid Overfitting User Specific Information in Federated Keyword Spottingcited by 4
  10. 2022FedorAS: Federated Architecture Search under system heterogeneitycited by 4
  11. 2022LiCo-Net: Linearized Convolution Network for Hardware-efficient Keyword Spottingcited by 4
  12. 2022BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to Real-Network Performancecited by 4
  13. 2022Improving Label-Deficient Keyword Spotting Through Self-Supervised Pretrainingcited by 3
  14. 2021Attention-Free Keyword Spottingcited by 2
  15. 2022Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attentioncited by 2
  16. 2023To Wake-up or Not to Wake-up: Reducing Keyword False Alarm by Successive Refinementcited by 2
  17. 2024EmoAttack: Utilizing Emotional Voice Conversion for Speech Backdoor Attacks on Deep Speech Classification Modelscited by 2
  18. 2024GE2E-KWS: Generalized End-to-End Training and Evaluation for Zero-shot Keyword Spottingcited by 2
  19. 2025SpikCommander: A High-performance Spiking Transformer with Multi-view Learning for Efficient Speech Command Recognitioncited by 2
  20. 2021Seed Words Based Data Selection for Language Model Adaptationcited by 1
  21. 2022Exploiting Hybrid Models of Tensor-Train Networks for Spoken Command Recognitioncited by 1
  22. 2022Improving Voice Trigger Detection with Metric Learningcited by 1
  23. 2022Fall Detection from Audios with Audio Transformerscited by 1
  24. 2023Small-footprint slimmable networks for keyword spottingcited by 1
  25. 2023Differentially Private Adapters for Parameter Efficient Acoustic Modelingcited by 1
  26. 2023EcoLearn: Optimizing the Carbon Footprint of Federated Learningcited by 1
  27. 2024Towards Federated Learning with On-device Training and Communication in 8-bit Floating Pointcited by 1
  28. 2025Large Language Model Data Generation for Enhanced Intent Recognition in German Speechcited by 1
  29. 2022Learning Audio Representations with MLPscited by 0
  30. 2022Boosting Tail Neural Network for Realtime Custom Keyword Spottingcited by 0
  31. 2022Transformer-based encoder-encoder architecture for Spoken Term Detectioncited by 0
  32. 2024Language-Universal Speech Attributes Modeling for Zero-Shot Multilingual Spoken Keyword Recognitioncited by 0
  33. 2024Advancing Airport Tower Command Recognition: Integrating Squeeze-and-Excitation and Broadcasted Residual Learningcited by 0
  34. 2024KWT-Tiny: RISC-V Accelerated, Embedded Keyword Spotting Transformercited by 0
  35. 2025Vocal Tract Length Warped Features for Spoken Keyword Spottingcited by 0
  36. 2025Noise-Agnostic Multitask Whisper Training for Reducing False Alarm Errors in Call-for-Help Detectioncited by 0
  37. 2025Gradient Norm-based Fine-Tuning for Backdoor Defense in Automatic Speech Recognitioncited by 0
  38. 2025SPBA: Utilizing Speech Large Language Model for Backdoor Attacks on Speech Classification Modelscited by 0

50 most cited of 59 All 59 (JSON)