mlsanityData to 7 Oct 2026

AST: Audio Spectrogram Transformer

arXiv 2104.01778 ↗First posted 5 Apr 2021cs.SD cs.AIcited by 634

Not on the map. Why?

Abstract (arXiv)

In the past decade, convolutional neural networks (CNNs) have been widely adopted as the main building block for end-to-end audio classification models, which aim to learn a direct mapping from audio spectrograms to corresponding labels. To better capture long-range global context, a recent trend is to add a self-attention mechanism on top of the CNN, forming a CNN-attention hybrid model. However, it is unclear whether the reliance on a CNN is necessary, and if neural networks purely based on attention are sufficient to obtain good performance in audio classification. In this paper, we answer the question by introducing the Audio Spectrogram Transformer (AST), the first convolution-free, purely attention-based model for audio classification. We evaluate AST on various audio classification benchmarks, where it achieves new state-of-the-art results of 0.485 mAP on AudioSet, 95.6% accuracy on ESC-50, and 98.1% accuracy on Speech Commands V2.

Key facts. Cited by 634 papers in the map (data through 7 Oct 2026). Builds on 20 of its 35 references that are in the map. First posted on arXiv 5 Apr 2021.

Builds on

20 of 35 references.

References · 20

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  12. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
8 more
  1. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  2. 2017Learning from Between-class Examples for Deep Sound Recognitioncited by 53
  3. 2020Rethinking CNN Models for Audio Classificationcited by 48
  4. 2020Streaming keyword spotting on mobile devicescited by 47
  5. 2020MatchboxNet: 1D Time-Channel Separable Convolutional Neural Network Architecture for Speech Commands Recognitioncited by 36
  6. 2020ESResNet: Environmental Sound Classification Based on Visual Domain Modelscited by 31
  7. 2019Sound Event Detection of Weakly Labelled Data with CNN-Transformer and Automatic Threshold Optimizationcited by 29
  8. 2020Training Keyword Spotters with Limited and Synthesized Speech Datacited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 19; 2022: 76; 2023: 106; 2024: 134; 2025: 169; 2026: 130.

634 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  3. 2021Attention Bottlenecks for Multimodal Fusioncited by 361
  4. 2021AudioCLIP: Extending CLIP to Image, Text and Audiocited by 335
  5. 2024InternVideo2: Scaling Foundation Models for Multimodal Video Understandingcited by 310
  6. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  7. 2022Wayformer: Motion Forecasting via Simple & Efficient Attention Networkscited by 279
  8. 2022Masked Autoencoders that Listencited by 271
  9. 2023WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Researchcited by 266
  10. 2021Efficient Training of Audio Transformers with Patchoutcited by 222
  11. 2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
  12. 2024Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilitiescited by 183
38 more
  1. 2022HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detectioncited by 174
  2. 2021SSAST: Self-Supervised Audio Spectrogram Transformercited by 166
  3. 2024GAMA: A Large Audio-Language Model with Advanced Audio Understanding and Complex Reasoning Abilitiescited by 151
  4. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  5. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  6. 2023Meta-Transformer: A Unified Framework for Multimodal Learningcited by 136
  7. 2025Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding and Expert Reasoning Abilitiescited by 127
  8. 2023EMAGE: Towards Unified Holistic Co-Speech Gesture Generation via Expressive Masked Audio Gesture Modelingcited by 101
  9. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 100
  10. 2023Joint Audio and Speech Understandingcited by 99
  11. 2022Learning Operators with Coupled Attentioncited by 88
  12. 2022FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generationcited by 87
  13. 2022MuLan: A Joint Embedding of Music Audio and Natural Languagecited by 85
  14. 2024Recent Advances in Speech Language Models: A Surveycited by 85
  15. 2022MAE-AST: Masked Autoencoding Audio Spectrogram Transformercited by 82
  16. 2025Vision-Language-Action Models for Robotics: A Review Towards Real-World Applicationscited by 82
  17. 2022Learning Audio-Video Modalities from Image Captionscited by 80
  18. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  19. 2023β-Variational autoencoders and transformers for reduced-order modelling of fluid flowscited by 76
  20. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  21. 2021MT3: Multi-Task Multitrack Music Transcriptioncited by 70
  22. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  23. 2024MLLM-Tool: A Multimodal Large Language Model For Tool Agent Learningcited by 66
  24. 2025MuQ: Self-Supervised Music Representation Learning with Mel Residual Vector Quantizationcited by 66
  25. 2023UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognitioncited by 64
  26. 2022Video Transformers: A Surveycited by 63
  27. 2021PolyViT: Co-training Vision Transformers on Images, Videos and Audiocited by 58
  28. 2022MAViL: Masked Audio-Video Learnerscited by 56
  29. 2022CLIPPO: Image-and-Language Understanding from Pixels Onlycited by 53
  30. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  31. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  32. 2021Sequence-to-Sequence Piano Transcription with Transformerscited by 48
  33. 2022Audiovisual Masked Autoencoderscited by 47
  34. 2024Foundation Models for Video Understanding: A Surveycited by 46
  35. 2023OmniVec: Learning robust representations with cross modal sharingcited by 44
  36. 2025Beyond Sight: Finetuning Generalist Robot Policies with Heterogeneous Sensors via Language Groundingcited by 40
  37. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
  38. 2023Emotional Speech-driven 3D Body Animation via Disentangled Latent Diffusioncited by 39

50 most cited of 634 All 634 (JSON)