mlsanityData to 7 Oct 2026

Study of positional encoding approaches for Audio Spectrogram Transformers

arXiv 2110.06999 ↗First posted 13 Oct 2021cs.SD cs.LG eess.AScited by 1

Not on the map. Why?

Abstract (arXiv)

Transformers have revolutionized the world of deep learning, specially in the field of natural language processing. Recently, the Audio Spectrogram Transformer (AST) was proposed for audio classification, leading to state of the art results in several datasets. However, in order for ASTs to outperform CNNs, pretraining with ImageNet is needed. In this paper, we study one component of the AST, the positional encoding, and propose several variants to improve the performance of ASTs trained from scratch, without ImageNet pretraining. Our best model, which incorporates conditional positional encodings, significantly improves performance on Audioset and ESC-50 compared to the original AST.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 30 references that are in the map. First posted on arXiv 13 Oct 2021.

Builds on

22 of 30 references.

References · 22

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  6. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  9. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2018Self-Attention with Relative Position Representationscited by 1,112
  12. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
10 more
  1. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  2. 2021AST: Audio Spectrogram Transformercited by 634
  3. 2021ConViT: Improving Vision Transformers with Soft Convolutional Inductive Biasescited by 484
  4. 2021Conditional Positional Encodings for Vision Transformerscited by 402
  5. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  6. 2020Streaming automatic speech recognition with the transformer modelcited by 73
  7. 2017Learning from Between-class Examples for Deep Sound Recognitioncited by 53
  8. 2021Audio Transformerscited by 41
  9. 2020A Sequential Self Teaching Approach for Improving Generalization in Sound Event Recognitioncited by 12
  10. 2018Learning Environmental Sounds with Multi-scale Convolutional Neural Networkcited by 4

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025The iNaturalist Sounds Datasetcited by 17