mlsanityData to 7 Oct 2026

Efficient Training of Audio Transformers with Patchout

arXiv 2110.05069 ↗First posted 11 Oct 2021cs.SD cs.LG eess.AScited by 222

Not on the map. Why?

Cited by 20 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The great success of transformer-based models in natural language processing (NLP) has led to various attempts at adapting these architectures to other domains such as vision and audio. Recent work has shown that transformers can outperform Convolutional Neural Networks (CNNs) on vision and audio tasks. However, one of the main shortcomings of transformer models, compared to the well-established CNNs, is the computational complexity. In transformers, the compute and memory complexity is known to grow quadratically with the input length. Therefore, there has been extensive work on optimizing transformers, but often at the cost of degrading predictive performance. In this work, we propose a novel method to optimize and regularize transformers on audio spectrograms. Our proposed models achieve a new state-of-the-art performance on Audioset and can be trained on a single consumer-grade GPU. Furthermore, we propose a transformer model that outperforms CNNs in terms of both performance and training speed. Source code: https://github.com/kkoutini/PaSST

Key facts. Cited by 222 papers in the map (data through 7 Oct 2026). Builds on 23 of its 30 references that are in the map. First posted on arXiv 11 Oct 2021.

Builds on

23 of 30 references.

References · 23

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  7. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  12. 2021AST: Audio Spectrogram Transformercited by 634
11 more
  1. 2019PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognitioncited by 563
  2. 2021Intriguing Properties of Vision Transformerscited by 446
  3. 2020FSD50K: An Open Dataset of Human-Labeled Sound Eventscited by 339
  4. 2020LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attentioncited by 326
  5. 2021Primer: Searching for Efficient Transformers for Language Modelingcited by 162
  6. 2019Adaptive Attention Span in Transformerscited by 148
  7. 2019Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answeringcited by 96
  8. 2021PSLA: Improving Audio Tagging with Pretraining, Sampling, Labeling, and Aggregationcited by 81
  9. 2020Acoustic scene classification in DCASE 2020 Challenge: generalization across devices and low complexity solutionscited by 48
  10. 2019The Receptive Field as a Regularizer in Deep Convolutional Neural Networks for Acoustic Scene Classificationcited by 22
  11. 2021Receptive Field Regularization Techniques for Audio Classification and Tagging with Deep Convolutional Neural Networkscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 39; 2024: 54; 2025: 64; 2026: 45.

222 papers.

Sort:

By area:

  1. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  2. 2023Simple and Controllable Music Generationcited by 480
  3. 2022BEATs: Audio Pre-Training with Acoustic Tokenizerscited by 378
  4. 2023AudioLDM 2: Learning Holistic Audio Generation with Self-supervised Pretrainingcited by 343
  5. 2023Make-An-Audio: Text-To-Audio Generation with Prompt-Enhanced Diffusion Modelscited by 323
  6. 2022AudioGen: Textually Guided Audio Generationcited by 292
  7. 2022Masked Autoencoders that Listencited by 271
  8. 2024Stable Audio Opencited by 192
  9. 2024Fast Timing-Conditioned Latent Audio Diffusioncited by 178
  10. 2022Contrastive Audio-Visual Masked Autoencodercited by 142
  11. 2024MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesiscited by 134
  12. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
38 more
  1. 2024Long-form music generation with latent diffusioncited by 88
  2. 2022HEAR: Holistic Evaluation of Audio Representationscited by 77
  3. 2025UniVerse-1: Unified Audio-Video Generation via Stitching of Expertscited by 58
  4. 2022MAViL: Masked Audio-Video Learnerscited by 56
  5. 2024EAT: Self-Supervised Pre-Training with Efficient Audio Transformercited by 52
  6. 2023A-JEPA: Joint-Embedding Predictive Architecture Can Listencited by 50
  7. 2024Masked Audio Generation using a Single Non-Autoregressive Transformercited by 50
  8. 2024Video-Guided Foley Sound Generation with Multimodal Controlscited by 49
  9. 2025HunyuanVideo-Foley: Multimodal Diffusion with Representation Alignment for High-Fidelity Foley Audio Generationcited by 42
  10. 2026MOVA: Towards Scalable and Synchronized Video-Audio Generationcited by 39
  11. 2022Supervised and Unsupervised Learning of Audio Representations for Music Understandingcited by 34
  12. 2024Foundation Models for Music: A Surveycited by 34
  13. 2024Tell What You Hear From What You See -- Video to Audio Generation Through Textcited by 34
  14. 2025ThinkSound: Chain-of-Thought Reasoning in Multimodal Large Language Models for Audio Generation and Editingcited by 33
  15. 2022XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learningcited by 30
  16. 2023Controllable Music Production with Diffusion Models and Guidance Gradientscited by 29
  17. 2023WavJourney: Compositional Audio Creation with Large Language Modelscited by 27
  18. 2022Efficient Large-scale Audio Tagging via Transformer-to-CNN Knowledge Distillationcited by 25
  19. 2023Self-supervised Audio Teacher-Student Transformer for Both Clip-level and Frame-level Taskscited by 25
  20. 2025KAD: No More FAD! An Effective and Efficient Evaluation Metric for Audio Generationcited by 24
  21. 2022Masked Modeling Duo: Learning Representations by Encouraging Both Networks to Model the Inputcited by 23
  22. 2024From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generationcited by 23
  23. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representationscited by 19
  24. 2023Fine-grained Audible Video Descriptioncited by 17
  25. 2024ETTA: Elucidating the Design Space of Text-to-Audio Modelscited by 17
  26. 2025OmniAudio: Generating Spatial Audio from 360-Degree Videocited by 17
  27. 2025SSLAM: Enhancing Self-Supervised Models with Audio Mixtures for Polyphonic Soundscapescited by 16
  28. 2023Sounding Video Generator: A Unified Framework for Text-guided Sounding Video Generationcited by 15
  29. 2023Contrastive Tuning: A Little Help to Make Masked Autoencoders Forgetcited by 15
  30. 2024What Do Language Models Hear? Probing for Auditory Representations in Language Modelscited by 15
  31. 2024Benchmarking Representations for Speech, Music, and Acoustic Eventscited by 14
  32. 2025Fast Text-to-Audio Generation with Adversarial Post-Trainingcited by 14
  33. 2022ASiT: Local-Global Audio Spectrogram vIsion Transformer for Event Classificationcited by 13
  34. 2023Zorro: the masked multimodal transformercited by 13
  35. 2024High Fidelity Text-Guided Music Editing via Single-Stage Flow Matchingcited by 13
  36. 2024LoVA: Long-form Video-to-Audio Generationcited by 13
  37. 2022CMKD: CNN/Transformer-Based Cross-Model Knowledge Distillation for Audio Classificationcited by 12
  38. 2023Masked Autoencoders with Multi-Window Local-Global Attention Are Better Audio Learnerscited by 11

50 most cited of 222 All 222 (JSON)